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EXECUTIVE SUMMARY 



For many years, policymakers have been concerned by the relatively low levels of academic 
achievement by economically disadvantaged K-12 students in math and science, by the 
underrepresentation of disadvantaged college students in math and science majors, and by the 
underrepresentation of people from disadvantaged groups in math and science careers. While 
racial gaps in math and science test scores narrowed somewhat in the 1970s and 1980s, 
substantial gaps persisted through the 1990s to the present. 

To help address these disparities, the U.S. Department of Education (ED) established a math 
and science initiative in 1990 within Upward Bound, a federal grant program designed to provide 
disadvantaged high school students with skills and experiences that will prepare them for college 
success. The initiative, referred to as Upward Bound Math-Science (UBMS), awards grants to 
institutions — largely colleges and universities — to operate UBMS projects. These projects were 
designed to differ from “regular” Upward Bound projects in several respects. To ensure that 
participants receive an intensive math and science precollege experience, UBMS projects 
provide instruction that includes hands-on experience in laboratories, computer facilities, and at 
field sites. Opportunities are also provided to leam from mathematicians and scientists 
employed at the host institution or engaged in research or applied science in other institutions in 
the community. A six-week summer program providing intensive instruction in laboratory 
science and mathematics through precalculus is also offered. 

Initially, ED funded 30 UBMS projects, by FY 2004, there were 127 UBMS projects 
serving 6,845 students at a total cost of $32.8 million. Therefore, the annual cost per student — 
approximately $4,800 — is comparable in cost to regular Upward Bound but much more 
expensive than other federally funded precollege programs. More than 80 percent of UBMS 
projects are hosted by four-year colleges and universities; most of the rest are hosted by two-year 
colleges (Curtin and Cahalan 2004). 

Participants in UBMS must meet the same eligibility requirements as regular Upward Bound 
students: students must (1) belong to families classified as low-income (taxable income of no 
greater than 150 percent of the poverty line), or (2) be a potential first-generation college student 
(neither parent has a bachelor’s degree). Some students who participate in UBMS summer 
programs are referred from regular Upward Bound programs and then return to those programs 
during the academic year. However, as would be expected, UBMS projects are more likely to 
consider students’ interests in math and science when reviewing applications than are most 
regular Upward Bound projects (Moore 1997b). While 25 percent of participants are white, most 
program participants are from underrepresented minority groups: about 60 percent of 

participants are African American or Hispanic (Curtin and Cahalan 2004). 



EVALUATION OF UPWARD BOUND MATH-SCIENCE 

Since 1991, Mathematica Policy Research, Inc., (MPR) has been conducting the National 
Evaluation of Upward Bound for ED. The centerpiece of this evaluation has been a random 
assignment evaluation of regular Upward Bound. In 1997, ED added a new component to the 
evaluation that is focused on UBMS. In 1998, MPR selected a random sample of the students 
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who participated in UBMS between 1993 and 1995 at projects that were still operating at that 
time. This report constitutes the first of two evaluation reports on UBMS, and it is based on 
participant surveys and student transcripts collected for this sample between 1998 and 1999 and 
again between 2001 and 2002. The second report is scheduled for completion in 2006 and will 
be based on data collected between 2003 and 2005. 

The evaluation of UBMS has two components: a descriptive analysis and an impact 

analysis. The descriptive analysis relies primarily on a survey of project directors to describe the 
resources available to UBMS projects; the types of institutions that host them; the credentials and 
demographic characteristics of project staff; recruitment, eligibility, and enrollment of students; 
student characteristics; and program offerings. The impact analysis is designed to measure the 
effects of UBMS on (1) performance in high school, especially in math and science courses; (2) 
postsecondary attendance, persistence and completion; and (3) the likelihood of completing a 
postsecondary degree in mathematics or a scientific field. 

The impact analysis is based on a comparison of UBMS participants with a sample of 
students that (1) applied to enroll in regular Upward Bound programs in the early 1990s, (2) 
never participated in UBMS and (3) have been tracked by MPR as part of the national 
evaluation. This comparison group was selected to ensure that it had similar characteristics to 
the sample of UBMS participants, and we controlled statistically for the small remaining 
differences in these characteristics between UBMS participants and the comparison group. 

If UBMS participants are more interested or skilled in math and science than the students in 
the comparison group, the estimated effects of the program may be subject to “selection bias” 
and may overstate the true effects of participating in UBMS. However, the comparison group 
we selected was probably the best available short of a randomized control group because the 
students in the comparison group exhibited the motivation to pursue Upward Bound services, 
and our analysis shows that the participant and comparison groups are similar in other ways as 
well. In addition, we implemented a data collection and analysis plan designed to minimize 
selection bias (see Chapter III for more details). While a control group from a randomized 
experiment would have prevented selection bias, the comparison group we selected greatly 
reduced the cost of the evaluation because we were already collecting data for this group as part 
of the national evaluation. 

Note that the descriptive findings and impact estimates presented in this report describe the 
operations and effects of the Upward Bound Math-Science Program as it operated in the mid- 
1990s. At that time, it was a relatively new program, and some changes have occurred in how 
UBMS projects operate. In Chapter II, we mention some of these changes as they are reflected 
in information provided to us by UBMS project directors in a survey of grantees. It is certainly 
possible that some of the changes in the program since the mid-1990s have influenced the 
effectiveness of UBMS projects, and the evaluation does not attempt to measure any changes in 
effectiveness since that time. In this report, we measure the effects of the program on people 
who participated between 1993 and 1995 and describe the operations of the program at that time. 
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REPORT FINDINGS 



From our descriptive analysis, we found that UBMS projects: 

• Provide a large quantity of academic instruction, in the summer, the average 
UBMS project provided a total of 240 hours of academic instruction, and 
participation in the program is roughly full-time for a six-week period. 

• Are most active during the summers. UBMS projects typically provide services, 
such as tutoring or study sessions, during the school year, but they provide most of 
their services during summer residential programs at the colleges or universities 
hosting the program. 

• Provide academic enrichment in math and science subjects. Many UBMS projects 
offer courses in algebra II, geometry, precalculus, biology, chemistry, physics and 
computer software; in contrast, few offer courses in Social Studies (though many 
offer English courses in addition to their math and science offerings). At most 
projects, the course work is designed to provide academic enrichment instead of 
academic remediation. 

• Provide instruction through a combination of single-subject courses and 
interdisciplinary instruction. While other instructional techniques were used, three 
out of four projects provided instruction primarily through single-subject academic 
courses or the combination of these courses with interdisciplinary instruction. 

Given the academic services provided by UBMS, it is natural to ask whether participating in 
UBMS affects the educational outcomes of the students that participate. From our impact 
analysis, we found that UBMS: 



• Improved high school grades in math and science and overall. UBMS had a 
positive effect on high school grades, increasing the average GPA in math courses 
from 2.7 to 2.8, the average GPA in science courses from 2.7 to 2.9 and the average 
GPA overall. 

• Increased the likelihood of taking chemistry and physics in high school. UBMS 
increased the likelihood that participants took upper-level science courses in high 
school, raising the percentage of students taking chemistry from 78 percent to 88 
percent and raising the percentage of students taking physics from 43 percent to 58 
percent, in contrast, UBMS did not affect coursetaking in advanced math subjects 
(see Chapter III, Exhibit III. 3). 

• Increased the likelihood of enrolling in more selective four-year institutions. 
UBMS increased the percentage of students that attended four-year colleges and 
universities from 71 percent to 82 percent. The increase in four-year attendance is 
particularly pronounced for more selective schools (those rated as “most selective”, 
“highly selective” or “very selective” by the Barron’s Guide): UBMS increased the 
percentage of students that attended more selective four-year colleges from 23 
percent to 33 percent (see Chapter III, Exhibit III.4). 
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• Increased the likelihood of majoring in math and science. UBMS affected 
students’ choice of major, increasing the percentage majoring (or planning to major) 
in math or science from 23 percent to 33 percent and decreasing the percentage 
majoring in a field outside of math or science and the social sciences from 51 to 42 
percent. UBMS also seems to increase the percentage of participants majoring in the 
social sciences (see Chapter III, Exhibit III. 6). 

• Increased the likelihood of completing a four-year degree in math and science. 
UBMS increased the percentage of students that earned a bachelor’s degree in a math 
and science field from 6 percent to 12 percent and decreased the percentage that 
earned a bachelor’s degree outside of math, science, and the social sciences from 20 
to 14 percent (see Chapter III, Exhibit III. 6). Because 47 percent of participants in 
our sample were still in college when we interviewed them in 2002, findings related 
to degree completion should be treated as preliminary, and a final assessment will be 
presented in a subsequent report. 

In addition, we computed separate impact estimates for subgroups defined by sex, race and 
ethnicity, and prior participation in regular Upward Bound. For some outcomes, we found 
differences in subgroup impacts that were statistically significant. For example, the effect of 
UBMS on four-year college attendance was larger for women than for men. However, the 
number of significant differences between subgroups was relatively small, and there was no 
obvious pattern to the findings suggesting that particular groups benefited more from UBMS 
than other groups. Therefore, it is not clear whether the significant subgroup differences are due 
to chance or to systematic differences in the effects of UBMS on different groups of participants. 

To summarize the report’s findings, UBMS provides intensive academic instruction in math 
and science, and our impact estimates suggest that it improves several student outcomes in high 
school and college. In addition, and consistent with the objectives of the program, preliminary 
estimates suggest that UBMS participation increases the odds of majoring in math or science. In 
the next report, we will reexamine the effects on college completion, examine the effects on 
labor market outcomes, such as employment in the sciences, and weigh the benefits of the 
program against the costs. 

It is tempting to compare the estimated impacts of UBMS to the estimated impacts of 
regular Upward Bound presented in earlier reports. However, it is important to recognize that 
the two studies used different methods: while the evaluation of regular Upward Bound is based 
on an experimental design, the “gold standard” in evaluation research, the evaluation of UBMS 
is based on nonexperimental methods that may suffer from selection bias, as described earlier. If 
the estimated effects of UBMS are inflated due to selection bias, then the impression based on 
our findings that UBMS is more effective than regular Upward Bound might be attributable to 
differences in the methods used to estimate the impacts instead of differences in the effectiveness 
of the two programs. 



xii 




I. INTRODUCTION 



For many years, policymakers have been concerned by the relatively low levels of academic 
achievement by economically disadvantaged K-12 students in math and science, by the 
underrepresentation of disadvantaged college students in math and science majors, and by the 
underrepresentation of people from disadvantaged groups in math and science careers. National 
statistics show that while the gaps between minorities’ 1 and whites’ math and science test scores 
narrowed somewhat in the 1970s and 1980s, gaps in test scores and other educational outcomes 
persisted through the 1990s to the present. 



• Disadvantaged students take fewer math and science courses in high school, in 
the 1991-92 school year, 57 percent of seniors in the lowest socioeconomic status 
(SES) quartile took a math course, compared with 75 percent of seniors from the 
highest SES quartile; 37 percent of seniors from the lowest SES quartile took a 
science course, compared with 61 percent of seniors from the highest SES quartile 
(U.S. Department of Education 1996b). in 1994, only 58 percent of black high 
school graduates had completed geometry while in high school, compared with 73 
percent of white high school graduates, in the same year, only 13 percent of black 
and Hispanic graduates had completed the common triad of science courses — 
biology, chemistry, and physics — compared with 23 percent of white graduates (U.S. 
Department of Education 1996a). 

• Minority college students are less likely to take math and science courses or earn a 
degree in math or science. Ten percent of black college students and 14 percent of 
Hispanics received credit for calculus or advanced math courses in the late 1980s, 
compared with 22 percent of whites. Sixteen percent of blacks and 2 1 percent of 
Hispanic college students earned course credits in chemistry, compared with 27 
percent of whites, and 8 percent of blacks and 1 1 percent of Hispanics earned college 
credit for physics, compared with 18 percent of white students (U.S. Department of 
Education 1994). Because minorities earned fewer college credits in math and 
science than whites, it is not surprising that they were less likely to earn degrees in 
those subjects. Black students earned 7 percent of all bachelor’s degrees in 1995-96, 
but just 7 percent of all bachelor’s degrees in math and science fields, in the same 

year, Hispanic students earned 5 percent of all bachelor’s degrees, but just 4 percent 

2 

of all bachelor’s degrees in math and science (U.S. Department of Education 1999). 



1 Ideally, socioeconomic measures such as income would be used to define groups, rather than race or 
ethnicity. For most education outcomes of interest, however, data are not presented on different income groups. 
Because racial and ethnic minorities are disproportionately lower-income (U.S. Census Bureau 2001:40), data based 
on race and ethnicity offer a reasonable, albeit imperfect, estimate of economically disadvantaged students’ 
educational experiences. 

2 The following subjects were classified as math or science: biological sciences and life sciences, computer 
and information sciences, engineering, engineering-related technologies, mathematics, and physical sciences and 
science technologies. 
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• Minorities are less likely than whites to enter careers in math and science. Among 
people who were working in a scientific field in 1995 and had obtained their college 
degree in the previous five years, only 6 percent were black (National Science 
Foundation 1995). However, in 1990, around the time those individuals would have 
been in college, blacks accounted for 14 percent of the U.S. population aged 18-24 
years old (Census Bureau 1990a, Census Bureau 1990b). 

A. UPWARD BOUND MATH-SCIENCE PROGRAM 

To help address these disparities, the U.S. Department of Education (ED) in 1990 
established the Upward Bound Math-Science Program (UBMS) within Upward Bound, a federal 
grant program designed to provide disadvantaged high school students with skills and 
experiences that will prepare them for college success. UBMS was designed to differ from 
“regular” Upward Bound in a few key respects. To ensure that participants receive an intensive 
math and science precollege experience, ED requires UBMS projects to provide instruction that 
includes hands-on experience in laboratories, computer facilities, and at field sites. Also 
provided are the following: opportunities to learn from mathematicians and scientists employed 
at the host institution or engaged in research or applied science in other institutions in the 
community; 3 involvement with tutors and counselors who are graduate and undergraduate math 
and science majors; and a six- week summer program consisting of daily course work and 
activities, instruction in laboratory science and mathematics through precalculus (in addition to 
foreign language, composition and literature, which are also required offerings at regular Upward 
Bound projects). 

Initially, ED funded 30 UBMS projects. By FY 2004, there were 127 UBMS projects 
serving 6,845 students at a total cost of $32.8 million. Therefore, the annual cost per student — 
approximately $4,800 — is comparable in cost to regular Upward Bound but much more 
expensive than other federally funded precollege programs. More than 80 percent of UBMS 
projects are hosted by four-year colleges and universities; most of the rest are hosted by two-year 
colleges (Curtin and Cahalan 2004). 

UBMS participants must meet the same eligibility requirements as regular Upward Bound 
participants: they must (1) come from families that are classified as low-income (taxable income 
not over 150 percent of the poverty line), or (2) be a potential first-generation college student 
(neither parent has a bachelor’s degree). Some students who participate in UBMS are referred 
from regular Upward Bound programs and then return to those programs during the academic 
year. However, as would be expected, UBMS projects are more likely to consider students’ 
interests in math and science when reviewing applications than are most regular Upward Bound 
projects (Moore 1997b). While 25 percent of participants are white, most program participants 
are from underrepresented minority groups: about 60 percent of participants are African 

American or Hispanic (Curtin and Cahalan 2004). 



; This requirement may have stemmed from concern that too much math and science instruction in high school 
is provided by teachers teaching out of their own field. 
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Despite coming from low-income families, the evidence suggests that on average, UBMS 
serves students who do well in high school and attend college at higher rates than the average 
low-income student. Data reported by Upward Bound projects suggest that prior to participating 
in Upward Bound, UBMS participants earned higher grades on average than regular Upward 
Bound participants (Curtin and Cahalan 2004). In addition, the national evaluation has shown 
that regular Upward Bound participants would have attended college at much higher rates than 
the average low-income student even if they had not participated in Upward Bound (Myers et al. 
2004). 4 Therefore, the evidence strongly suggests that UBMS serves high school students who 
are much more likely to attend college than the average low-income student. 



B. EVALUATION OF THE UPWARD BOUND MATH-SCIENCE PROGRAM 

The legislation establishing Upward Bound authorizes ED to sponsor studies of it, including 
examinations of program effectiveness. In 1991, ED awarded a contract to Mathematica to 
conduct the National Evaluation of Upward Bound. This evaluation has several components, but 
its signature feature is an experiment to measure the effects of participating in regular Upward 
Bound. We selected a random sample of Upward Bound projects (excluding UBMS projects); 
for each of these projects, we randomly assigned eligible applicants to a treatment group, which 
was offered the chance to participate in the program, or a control group, which was not. The 
evaluation is ongoing, and it was one of the first to use experimental methods to measure the 
effects of a federally funded education program. 

This report presents the results of an evaluation of the Upward Bound Math-Science 
Program. In 1997, Mathematica completed two reports on UBMS. One provided a descriptive 
analysis of the program based primarily on site visits to a representative sample of 14 UBMS 
projects (Moore 1997a). The other provided an assessment of the feasibility of conducting a 
rigorous evaluation of the effects of UBMS on student outcomes (Myers 1997). When ED 
awarded a contract to Mathematica in 1997 to extend its evaluation of the effects of regular 
Upward Bound, it also specified an evaluation of the effects of UBMS. This evaluation consists 
of two components: a descriptive analysis and an impact analysis. The descriptive analysis 
relies primarily on a survey of UBMS project directors conducted in the spring of 1998. The 
analysis is designed to describe the resources available to UBMS projects; the types of 
institutions that host them; the credentials and demographic characteristics of project staff; 
recruitment, eligibility, and enrollment of students; student characteristics; and a description of 
the program, including its goals, academic orientation, instructional methods and the intensity 
and quantity of the services provided. 

The UBMS impact study is designed to measure the effects of participating in UBMS on 
college enrollment, choice of major, and other outcomes for students who participated during the 



4 While the last report from the national evaluation shows that about 70 percent of regular Upward Bound 
participants would have attended a postsecondary institution even if they had not participated in Upward Bound 
(Myers et al. 2004), only 53 percent of 1992 high school graduates from the lowest SES quartile attended a 
postsecondary institution by 1994 (U.S. Department of Education 1997). 
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summer of 1993, 1994, or 1995. 5 Conceptually, the study contrasts how participants fared with 
how they would have fared if they had not participated in UBMS. We compared UBMS 
participants with eligible applicants to the regular Upward Bound projects participating in the 
national evaluation. From this pool, we systematically selected a matched comparison group of 
students who were as similar as possible to UBMS participants in terms of characteristics and 
experiences that could potentially predict later outcomes. These characteristics included 
demographics — such as sex, race, and ethnicity — and prior academic achievement such as grade 
point average and math and science courses taken in 9th grade. The key difference was that the 
matched comparison students did not participate in UBMS. 

The selection of matched comparison students also took into account experiences in other 
precollege programs, in particular regular Upward Bound. Because regular Upward Bound is an 
intensive program that can influence high school achievement and postsecondary outcomes 
(Myers and Schinn 1999; Myers et al. 2003), it is important to account for exposure to regular 
Upward Bound when estimating how UBMS participants would have fared if they had not 
participated in UBMS. For UBMS participants who had previously participated in a regular 
Upward Bound program — perhaps during the academic year — we selected comparison students 
who had also participated in the regular Upward Bound. 6 For UBMS participants who had not 
participated in regular Upward Bound, we selected comparison students who did not participate 
in regular Upward Bound. 7 

Several data sources play a key role in the impact analysis. Baseline characteristics were 
collected for comparison group members through the baseline survey for the evaluation of 
regular Upward Bound; baseline information on many of the same characteristics was collected 

o 

for UBMS participants through a follow-up survey conducted in 1999. This follow-up survey 
was used to collect information about educational outcomes for UBMS participants, and a 
similar survey was used to collect analogous information for comparison students. Finally, 
secondary and postsecondary transcripts were collected for both types of students to assess 
academic achievement. 



C. OVERVIEW OF THIS REPORT 

The remainder of this report is organized as follows. Chapter II describes the operation of 
the UBMS program. Chapter III presents findings from the impact analysis. 



5 Because the sample was not selected until 1998, we restricted the sample to participants at UBMS projects 
that were still operating that year: obtaining lists of participants from programs that were no longer operating in 
1998 would have been nearly impossible. 

6 These comparison students were selected from the treatment group for the evaluation of regular Upward 
Bound. For a more thorough discussion of how the treatment group was selected, see Myers et al. (2004), Appendix 
A. 



7 These comparison students were selected from the control group for the evaluation of regular Upward Bound. 
For a more thorough discussion of how the control group was selected, see Myers et al. (2004), Appendix A. 

s While the 1999 survey was conducted four to six years after our sample had participated in the program, most 
of the baseline information collected — including sex, race, and ethnicity — is time-invariant. 
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II. THE OPERATION OF THE UPWARD BOUND MATH-SCIENCE PROGRAM 



To interpret information on the impacts of UBMS, it is necessary to understand what the 
program entails. This chapter describes key features of the operations of UBMS projects, 
including the characteristics of host institutions and staff, projects’ recruitment practices and 
enrollment levels, participants’ characteristics and projects’ goals and services. For context, this 
chapter presents comparable infonnation on the operations of regular Upward Bound when 
possible. 

The primary data source for this chapter is a survey of UBMS projects conducted in the 
spring of 1998. The survey sample consisted of all 81 projects operating at the time, and 74 of 
the 81 projects responded to the survey. 9 ’ 10 the survey requested information about program 
operations in two separate years — (1) 1994 (in the middle of the period over which our sample 
was participating in UBMS) and (2) 1998 (the year prior to the survey) — but some questions 
were specific to 1998. When possible, we focus our analysis of program operations on 1994 to 
facilitate comparisons with regular Upward Bound projects operating in 1993, as reported in 
Fasciano and Jacobson (1997) and to describe the programs that served the same cohorts of 
participants for whom we measured the impacts of the program (see Chapter III). 11 To augment 
the information provided by the survey of UBMS projects, we also use infonnation from case 
studies and annual perfonnance reports (Moore 1997b). 

The findings in this chapter indicate that UBMS projects provide intensive academic 
enrichment to disadvantaged high school students in math and science using staff with strong 
academic credentials in those subjects. Some of the features that make UBMS projects 
distinctive, even from regular Upward Bound projects, are: (1) high levels of annual funding per 
student and low student-teacher ratios, (2) recruiting strategies that attract students from wide 
geographic areas, (3) service provision that is heavily concentrated in residential programs 
during the summer, (4) course offerings that focus on math and science relative to other subjects, 
(5) academic preparation over nonacademic college preparatory activities, and (6) academic 
enrichment over remediation. The remainder of the chapter provides a description of UBMS and 
an assessment of its distinctive features. 



9 We did not adjust (weight) for survey nonresponse, reasoning that the number of nonrespondents was low 
enough to eliminate any serious concerns about data representativeness. Also, rarely did more than three UBMS 
projects fail to respond to any particular item on the questionnaire. 

10 We excluded the one project that reported serving only veterans in 1998. Note that veterans’ projects were 
also excluded from the survey of regular Upward Bound grantees, so the comparisons that are made in this chapter 
between the two types of Upward Bound programs are based on Upward Bound projects that did not exclusively 
serve veterans. 

11 Unless noted otherwise, the results for 1998 were generally similar to those for 1994. 
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A. PROJECT HOSTS AND STAFF 



The impacts of UBMS projects on student outcomes may depend on the types of institutions 
that host them and the people they hire to serve as instructors and other staff. In this section, we 
describe the types of institutions that host UBMS projects and the staff that provide services to 
program participants. 



1. Host institutions: Two- and Four-Year Colleges and Universities 

The types of institutions that host a UBMS project may influence where students attend 
college. Most Upward Bound programs are hosted by either two- or four-year postsecondary 
institutions. Evidence from the national evaluation of regular Upward Bound suggests that 
participation at projects hosted by four-year colleges raises the probability of attending a four- 
year college, and participation at projects hosted by two-year colleges raises the probability of 
attending a two-year college (Myers et al. 2002b). Therefore, the types of institutions that host 
UBMS projects may influence the types of postsecondary institutions that program participants 
subsequently attend. 

Nearly nine out of ten UBMS projects operating in the mid-1990s were hosted by four-year 
colleges, a substantially higher proportion than among regular Upward Bound projects (see 
Exhibit II. 1). Four-year colleges may find it easier than other potential host institutions to meet 
some of ED’s guidelines for UBMS, including offering hands-on experience in laboratories and 
computer facilities, opportunities to learn from mathematicians and scientists engaged in 
research or applied science and involvement with tutors and counselors who are graduate and 
undergraduate students in math and science. 



2, Summer Program Staff and Project Director 

UBMS projects are directed by highly educated individuals and staffed by people with 
strong credentials in math and science. These staff have responsibility for a relatively small 
number of students, which may provide opportunities for individual instruction. At the typical 
project, the project director and staff can provide same -race role models for many of the students 
they serve. The sections below provide more detail on our findings concerning staff size and 
composition by job title, staff credentials, and the racial composition of project staff. 



a. Staff Size and Composition by Job Title 

In 1994, UBMS projects had an average of 24 staff members, comprising roughly eight 

instructors, five resident counselors, four mentors, three tutors, two administrators, one academic 

12 

or guidance counselor and one clerical staff member. ~ Overall, the average student-staff ratio in 
summer 1 998 was 2:1, with a range from about 1 : 1 to 5 : 1 . 



12 By 1998 the average MSC had almost 26 staff, including 9 instructors. 
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Exhibit II. 1 



Types of Institutions that Hosted UBMS Projects, 1995 



Type of institution 


Upward Bound Math-Science 


Regular Upward Bound 


Four-year college or university 


88% 


68% 


Two-year college 


11 


28 


Other institution 


2 


4 



Note: Percentages may not sum to 100 percent due to rounding. 

Source: Moore 1997b, Exhibit II. 1, p. 15. 



These findings, combined with findings from Moore (1997b), suggest that student-staff 
ratios are typically lower in UBMS projects than in regular Upward Bound projects. The survey 
of grantees did not collect infonnation on the number of full-time-equivalent (FTE) staff, but the 
information available suggests that UBMS projects typically maintain student-staff ratios that are 
substantially lower than in regular Upward Bound. Moore (1997b) found that 14 randomly 
selected UBMS projects visited in summer 1996 had an average of 2.6 students per FTE staff, 
including administrators, and 8.2 students per FTE instructional staff (Moore 1997b, Exhibit 
II. 6). In contrast, tabulations from the data used by Fasciano and Jacobson (1997) indicate that 
in summer 1992, regular Upward Bound projects had more students per staff member — 5.1 
students per FTE staff and 13.6 students per FTE instructional staff (Moore 1997b, Exhibit II. 6). 



b. Credentials 

At the average UBMS project in 1998, most staff were highly educated and had educational 
backgrounds in math and science. About one-quarter had attended some college but not obtained 
a degree, another quarter had obtained a bachelor’s or associate’s degree (mostly bachelor’s 
degrees) and the rest had done graduate work or obtained a graduate degree. Most staff members 
without undergraduate degrees were undergraduate students who served as UBMS mentors, 
tutors and resident counselors while working toward bachelor’s degrees in math, science, or 
education. Approximately two out of five staff members had their highest degree in science or 
the social sciences (31 percent) or math (10 percent); additionally, most staff members without a 
degree were working toward a bachelor’s degree in math, science, or education. 

Most instructors at the average UBMS project had experience teaching math or science. 
During the school year, most instructors were either high school teachers (41 percent) or 
postsecondary teachers (31 percent); one-fifth were graduate students (14 percent) or 
undergraduates (6 percent). Moreover, at the typical project, two-thirds of the high school 
teachers and three-quarters of the postsecondary teachers taught in a math or science field. 

The professional and educational backgrounds of UBMS project directors provide insight on 
their credentials to direct projects. One-fifth of the directors were faculty members at the host 
institution or another college, roughly the same percentage as in regular Upward Bound. In 
1998, two-thirds of the directors held a master’s degree and one-fifth held a doctorate. About 
half had their highest degree in education, and less than one-fifth had their highest degree in 
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engineering, mathematics, or physical sciences. Although UBMS project directors were less 
likely than program staff as a whole to have a background in math or science, this may not be 
surprising since subject area expertise is probably less important for administrators than for other 
staff. Compared with regular Upward Bound project directors, UBMS project directors were 
more highly educated. They were, for example, twice as likely to have a doctorate. 



c. Race and Ethnicity 

At about 9 out of 10 UBMS projects operating in 1998, one racial or ethnic group accounted 
for a majority of the staff (see Exhibit II. 2). In many cases this pattern may have reflected a 
conscious strategy, also used in regular Upward Bound, to provide minority students with same- 
race role models. For example, at 21 UBMS projects, a majority of the staff members were 
black; at 18 of these, a majority of the students were also black. 

The racial and ethnic profile of UBMS project directors was similar to that of UBMS project 
staff (see Exhibit II. 3). Both staff and project directors were nearly evenly split between white 
and nonwhite; project directors were slightly more likely to be black and less likely to be Asian 
than other staff. The race and ethnicity of the UBMS project director often matched that of the 
predominant student racial and ethnic group: more than three-fourths of the minority directors 
headed programs where students from the same group constituted a plurality of participants. 



B. ELIGIBILITY, RECRUITMENT AND ENROLLMENT, AND STUDENT 
CHARACTERISTICS 

To shed light on the types of students that participate in UBMS, we examine the eligibility 
criteria that students must meet and the recruiting strategies that UBMS projects use to attract 
students. We also examine the characteristics of participants as reported by project directors. 



1. Eligibility 

UBMS projects have to meet the same federal rules as regular Upward Bound projects 
concerning the composition of participants. At each project, at least two-thirds of the 
participants must be both low-income and potential first-generation college students; the 
remaining students must meet either of these two criteria. At the average project in summer 
1998, about 77 percent of students met both of these eligibility criteria, about 14 percent were 
first-generation only and about 9 percent were low-income only, very similar to the distribution 
in regular Upward Bound during 1992-93. In addition, UBMS and regular Upward Bound 
projects are only allowed to serve students who have completed 8th grade. 



13 An additional 12 percent had their highest degree in the social sciences. 
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Exhibit 11.2 



Predominant Racial and Ethnic Group for UBMS Staff, Summer 1998 



Majority white 
54% 




Note: Percentages may not sum to 100 percent due to rounding. The 12 percent of projects with no 

majority racial and ethnic group included 6 percent of projects with a plurality of blacks and 4 
percent with a plurality of whites; the remainder had no plurality. 



Source: 1998 survey of UBMS projects. 



Exhibit II. 3 

Racial and Ethnic Distribution for UBMS Staff and Project Directors, Summer 1998 



Race and Ethnicity 


Staff 


Director 


White 


49% 


49% 


Black 


33 


39 


Hispanic or Latino 


7 


7 


Asian 


4 


1 


American indian or Alaskan Native 


2 


4 



Note: Percentages may not sum to 100 percent because (1) people may fall into multiple categories, (2) the 

Pacific Islander category was excluded from the exhibit because only nine staff members nationwide fell 
into this category and (3) some staff members may not have been classified by race or ethnicity. 



Source: 1998 survey of UBMS projects. 
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Most UBMS projects also adopt additional student eligibility criteria for enrollment in the 
program — for example, requirements about grade level, school course work, or 
recommendations. In 1994, over three-fourths of projects required students to have finished 9th 
grade; a few projects required 10th grade completion. In addition, nearly all UBMS projects 
required a teacher recommendation and completion of at least one high school course in math or 
science, and applicants enrolled in regular Upward Bound commonly needed a recommendation 
from the director. Finally, about 30 percent of UBMS projects prohibited students from 
returning from the previous summer’s program, and almost half prohibited students from 
returning unless they met certain criteria. 14 



2, Recruitment and Enrollment 

To find a pool of potentially eligible applicants, UBMS projects focused mainly on other 
precollege programs or secondary schools (see Exhibit II. 4). Among UBMS projects operating 
in summer 1994, nearly all recruited from regular Upward Bound projects, while substantial 
majorities also recruited from Talent Search projects and from middle or high schools directly. 
However, it is important to note that while almost all UBMS projects recruited from regular 
Upward Bound projects in 1994, data from the evaluation suggest that fewer than one in five 
UBMS participants had previously participated in regular Upward Bound. 

Historically, UBMS projects have cast a wide net in recruiting students beyond the local 
areas of the host institutions. In 1994, only about one in ten UBMS projects recruited 
exclusively from a specific and typically local set of feeder schools or Upward Bound projects; 
the rest recruited from state-wide or regional lists of schools and programs (see Exhibit II. 5). By 
1998, however, the percentage of UBMS projects that recruited exclusively from a specific set of 
feeder schools or Upward Bound projects had tripled. Therefore, it appears that over time, more 
UBMS projects are taking local recruitment strategies like regular Upward Bound projects. 

By design, UBMS projects are smaller than regular Upward Bound projects. Through 
recruitment, UBMS projects received an average of 108 applications for the summer of 1994, 
ranging from a low of 50 to a high of 300, and they enrolled between 40 and 53 students. In 
contrast, regular Upward Bound programs enrolled an average of about 75 students in the mid- 
1990s (Moore 1997b). 



14 These practices changed dramatically by 1998, when only 7 percent of UBMS projects prohibited all 
students from returning from the prior summer, and 7 1 percent prohibited students from returning unless they met 
certain criteria. The questionnaire did not address the specific types of criteria that projects imposed. 
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Exhibit 11.4 



Targets for Recruiting by UBMS Projects, 1994 




0% 20% 40% 60% 80% 100% 

Percent of Upward Bound Math-Science Projects 



Source: 1 998 survey of UBMS projects. 



Exhibit 11.5 

Geographic Scope of Recruiting by UBMS Projects, 1994 



Both 

45% 




Note: Percentages may not sum to 100 percent due to rounding. 

Source: 1998 survey of UBMS projects. 
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3. Student Characteristics 



In our 1998 survey of UBMS projects, project directors provided information on the 
distribution of students participating in their projects by sex, race, grade level, and place of 
residence: 15 



• Sex. At the average UBMS project, like the average regular Upward Bound project, 
60 percent of students were female. 16 However, this varied considerably across 
UBMS projects from a low of 25 percent to a high of 78 percent. 

• Race. On average, UBMS projects served an ethnically diverse group of students: 
42 percent black, 27 percent white, 15 percent Hispanic, 8 percent Asian, 5 percent 
American indian and 1 percent Native Hawaiian or other Pacific Islander. However, 
most UBMS projects serve participants where one racial and ethnic group 
constituted a majority (see Exhibit II. 6). 17 Furthermore, some UBMS projects 
exclusively served students from a single racial or ethnic group. For example, six 
UBMS projects reported that all of its participating students were black. 

• Grade level. The eligibility guidelines discussed above, along with other factors, 
can affect the distribution of students across different grade levels. At the average 
project, 29 percent of participants were entering 12th grade, 37 percent were 
entering 11th grade, 27 percent were entering 10th grade and 6 percent were 
entering 9th grade. These exhibits suggest that on average, UBMS projects serve 
students who are slightly closer to graduation than is the case at regular Upward 
Bound projects. However, there was substantial variation in the grade level 
distribution of participants across projects in 1998. for example, one UBMS project 
reported that all of its participants were entering 12th grade, while four reported that 
none were rising seniors. 19 

• Place of residence. Given that most projects recruited across the state or region, it 
is not surprising that many UBMS participants came from outside the grantee’s local 
city or town. At the average project, only about 25 percent of the students were 
locals. As we would expect, projects that recruited only from a set of local schools 
or regular Upward Bound projects served considerably higher percentages of 
students from the local area than other UBMS projects. 



15 Most of these questions were focused on 1998 participants, so the analysis in this section is focused on 1998 
instead of 1994. 

16 Data based on all UBMS projects operating in 1998 and all regular Upward Bound projects operating in 

1992. 



17 In comparison, at 87 percent of regular Upward Bound projects operating in 1992-93, one racial and ethnic 
group accounted for a majority of participants. 

18 In regular Upward Bound during the summer of 1992, 20 percent of participants were entering 12th grade, 
32 percent were entering 1 1th grade, 31 percent were entering 10th grade, and 16 percent were entering 9th grade. 

19 In addition, the proportion of rising juniors ranged from 0 to 72 percent, and the proportion of rising 
sophomores ranged from 0 to 60 percent. 
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Exhibit 11.6 



Predominant Racial and Ethnic Group for Participants at UBMS Projects, 

Summer 1998 



Majority white 




Note: Percentages may not sum to 100 percent due to rounding. The 37 percent of UBMS projects 

with no majority racial and ethnic group of students included 17 percent with a plurality of white 
students, 11 percent with a plurality of Hispanics, 6 percent with a plurality of blacks, 3 percent 
with a plurality of American indians and 1 percent with a plurality of Asians. 

Source: 1998 survey of UBMS projects. 



Moore (1997b) described UBMS participants as a more select group than regular Upward 
Bound participants based on having earned somewhat higher grades and having greater interest 
in math and science (pp. 23, 26) prior to participating in Upward Bound. Discussions with 
UBMS and regular Upward Bound staff revealed that UBMS participants were typically 
considered “more serious about school” than regular Upward Bound participants (Moore 1997b, 

p. 26). 

C. PROGRAM DESCRIPTION 

In a college-like setting, UBMS projects offer academic enrichment in math and science to 
improve student achievement in those subjects and expose students to math and science careers. 
In this section, we describe the following features of UBMS projects in more detail: the setting 
in which these projects provide services; the goals, academic orientation, academic offerings, 
and instructional approaches of these projects; and the intensity and quantity of services the 
UBMS projects provide. 
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1. Setting 



As described earlier, UBMS projects are typically hosted by two- and four-year 
postsecondary institutions (see Section A.l). Most UBMS projects are hosted by four-year 
colleges and universities, and most of these institutions have dormitories to house their students. 
These dormitories are often available in the summers to house participants of summer programs 
hosted by these institutions. 

UBMS projects typically exposed participants to a college setting during the summer 
program by housing them in the college dormitories. Virtually all the UBMS projects we 
surveyed (100 percent in 1994, 97 percent in 1998) offered a residential component to their 
summer programs, compared with 87 percent of regular Upward Bound programs in 1992 
(Moore 1997a). At almost all UBMS projects, students lived in the dormitories for the entire 
summer program, which lasted about six weeks on average. Therefore, for six weeks, 
participants lived on campus like many college students do during the academic year. 



2. Goals of the Program 

As mentioned in Chapter I, the general objective of the Upward Bound Math-Science 

program is to prepare participating students for postsecondary programs leading to careers in 

math and science. Seven out of ten UBMS projects operating in 1994 rated “academic 

performance in math and science” as their most or second most important goal (see Exhibit II. 7). 

The focus on academic improvement was similar to the focus of regular Upward Bound projects 

21 

operating around the same time." 

However, two goals that regular Upward Bound projects considered moderately important 
were not considered important by UBMS projects. First, only 13 percent of UBMS projects 
reported that one of their top two goals was fostering students’ personal skills (e.g., goal 
orientation, ability to adapt to new settings), compared with 31 percent of regular Upward Bound 
programs. Second, none of the UBMS projects cited improving students’ access to financial aid 
as one of their top two goals, compared with 35 percent of regular programs. 



3. Academic Orientation 

UBMS projects try to provide academic enrichment beyond what students are exposed to in 
school (see Exhibit II. 8). Very few projects emphasized remedial instruction in 1994. 22 While 



20 Only 4 percent of UBMS projects in 1994 had a residential component shorter than the summer program, but 
by 1998, the rate had increased to 1 1 percent. 

21 Eighty-seven percent of regular Upward Bound projects rated “academic improvement” as their most or 
second most important goal in 1993. If the regular Upward Bound grantee survey had also listed “academic 
improvement in math and science,” it is possible that some respondents would have cited that as one of their top two 
goals: Fasciano and Jacobson (1997) characterized 37 percent of regular Upward Bound projects as having a strong 
emphasis on math and science. 

~ 2 By 1998, the relative focus on enrichment was even greater, with 83 percent of UBMS projects citing 
enrichment as a major emphasis and only 1 percent citing remediation. 
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Exhibit 11.7 



Most Important Goals of UBMS Projects, Summer 1994 



Academic improvement 
in math and science 

Exposure to careers in 
math and science 

Perseverance in math 
and science 

General academic 
improvement 



Exposure to college 




0% 20% 40% 60% 80% 100% 

Percent of UBMS Projects 



■ Most important D Second most important 



Source: 1 998 survey of UBMS projects. 



Exhibit II. 8 

Academic Orientation of UBMS Projects 3 



Major Emphasis 


UBMS 


Regular Upward Bound 


Summer 1994 


Summer 1992 b 


Support — instruction that parallels what students are 
taught in regular school courses 


33% 


55% 


Remediation — instruction that concentrates on 
fundamental concepts and skills that were taught in 
earlier grades 


6 


23 


Enrichment — instruction in concepts and material 
beyond what students are exposed to in regular school 
classes 


73 


69 



a Percentages do not sum to 100 because grantees were allowed to rate more than one approach as a major emphasis. 
b Excludes summer bridge programs for Upward Bound participants who have just graduated from high school. 

SOURCES: 1998 survey of UBMS projects, Fasciano and Jacobson 1997. 
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about one in four regular Upward Bound programs emphasized the provision of remedial 
instruction, fewer than one in ten UBMS projects reported doing the same. 

The focus of UBMS on academic enrichment over remediation is consistent with the types 
of students served by the program. As described earlier, findings in Moore (1997b) indicate that, 
on average, UBMS participants probably had less need for remedial support than regular Upward 
Bound participants. 



4. Summer offerings 

In accordance with program guidelines, UBMS projects offered instruction in a diverse array 
of academic subject areas (see Exhibit II. 9). Seventy-five percent or more of these projects in 
1994 offered instruction in the following subjects: writing and composition, algebra II, 

geometry, precalculus, computer applications and software use, biology, chemistry and physics. 
The average total number of offerings in 1994 was about 14, with a range of 2 to 22. The 
average number of offerings in math and science was about 7, with a range of 1 to 1 1 . 23 Thus, 
on average, math and science courses accounted for roughly half of UBMS projects’ total 
offerings. 

UBMS projects clearly differed from regular Upward Bound projects in their relative 
emphasis on certain subjects. First, as expected, they concentrated their offerings more on math 
and science. Although UBMS projects were no more likely than regular Upward Bound projects 
to offer certain math or science courses (for example, algebra II, geometry precalculus, calculus, 
biology and chemistry), UBMS projects were much less likely to offer instruction in areas 
outside of math and science, such as social science or history courses or electives or non- 
academic courses. Second, consistent with their greater emphasis on enrichment than on 
remediation, UBMS projects were less likely than regular Upward Bound programs to offer low- 
end courses such as reading comprehension and vocabulary, pre-algebra, and earth science (see 
Exhibit II.9). 

To help prepare students for a postsecondary education and post-collegiate careers in math 
or science, UBMS projects also offered a range of support services and activities (see Exhibit 
II. 10). Among the most common activities were field trips (e.g., to math or science facilities) 
and assistance with college and financial aid applications. The average number of these non- 
instructional offerings in 1994 was about 10, with a range of 3 to 15. 

UBMS projects were substantially less likely than regular Upward Bound projects to offer 
services focused on preparing for college. Many regular Upward Bound projects would have 
provided these services during the academic year: both regular Upward Bound and Math 

Science programs focus on academics during the summer. Because many UBMS participants 
participated in other precollege programs during the academic year, UBMS project staff could 



23 It is not clear why some UBMS projects apparently had such a small number of instructional offerings. One 
possibility is that their instructional approach centered on interdisciplinary courses or large projects that covered 
multiple subject areas. However, only 3 of the 12 projects that reported five or fewer math and science offerings in 
1994 also reported that their primary instructional method was interdisciplinary courses or large projects and 
experiments. A second possibility is underreporting of instructional offerings by project directors. 
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Exhibit II. 9 



Instruction Offered by UBMS Projects, by Subject Area 2 





UBMS 
Summer 1994 


Regular Upward Bound 
1992 b 


English/Language Arts 


Writing/Composition 


93% 


100% 


Literature 


60 


83 


Reading Comprehension and Vocabulary 


65 


98 


English as a Second Language 


13 


11 


foreign Language 


54 


35 


Other 


9 


13 


Mathematics 


Pre-Algebra 


36 


82 


Algebra I 


69 


96 


Algebra II 


81 


95 


Geometry 


80 


95 


Precalculus 


80 


80 


Calculus 


52 


58 


Statistics 0 


17 


C 


Trigonometry 0 


7 


C 


Other 


9 


24 


Computers 


Programming 


43 


47 


Applications/Software Use 


85 


79 


internet/Web Page Design 0 


7 


C 


Other 


7 


6 


Science 


Physics 


76 


63 


Biology 


87 


89 


Chemistry 


81 


81 


Earth Science 


48 


66 


Other 


15 


19 


Social Science/History 


History 


11 


47 


Geography 


9 


24 


Sociology 


4 


17 


Psychology 


8 


15 


Government/ Civics 


9 


40 


Other 


8 


13 


Electives/Non-Academic Courses 


Performing Arts 


31 


53 


Art 


26 


53 


Journalism 


28 


52 


Speech/Public Speaking 


48 


59 


Physical Fitness 


56 


69 


Other 


6 


26 



a UBMS projects offer instruction in many areas besides math and science, either to meet regulatory requirements or 
simply to ensure that their program will interest and benefit students in many ways. 
b 1992 non-bridge summer programs or 1992-93 academic year. 

c Neither survey listed statistics, trigonometry, or internet or Web page design, but enough project directors specified 
them under “other courses” that we present data on these courses separately. 

Sources: 1998 survey of UBMS projects, Fasciano and Jacobson (1997:39). 
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Exhibit II. 10 



Noninstructional Services Offered by UBMS Projects 





UBMS 
Summer 1994 


Regular Upward Bound 
1992 a 


College preparation/skills 


Campus visits 


74% 


98% 


Adjusting to college living 


98 


92 


ACT/SAT preparation 


65 


97 


PSAT/PLAN or PACT preparation 


24 


73 


Help with financial aid or scholarships' 5 


80 


100 


Assistance with college applications 


78 


99 


Assistance with financial aid applications 


72 


100 


Career/employment assistance 


Site visits to employers 0 


65 


59 


On-campus (employers or career representatives) 


63 


78 


Project-related work experience 




49 d 


JTPA job 


0 


d 


Work- study job 


4 


d 


Math or science internships 


24 


d 


Job through other partnerships 


7 


d 


Field trips to 


Academic science or math facilities 


98 


e 


Non-academic science or math facilities 


94 


e 


Conduct math- or science-related field work 


85 


e 


Other 


93 


e 



a 1992 nonbridge summer programs or 1992-93 academic year. 

b In the regular Upward Bound grantee survey, this item was phrased, “Identify sources of financial aid.” 
c In the regular Upward Bound grantee survey, this item was phrased, “Site visit to employers or job shadowing.” 
"'Although the regular Upward Bound grantee survey had three separate items about JTPA, work-study and other 
partnerships, the results were reported only in the aggregate, and it did not ask about math or science internships 
(Fasciano and Jacobson 1997:54). 

"the regular Upward Bound grantee survey asked about field trips of varying lengths, not the destinations. 
Sources: 1998 survey of UBMS projects, Fasciano and Jacobson (1997:54). 



have reasonably expected that those other programs were assisting students in preparing for 
college. 



5. Academic-Year offerings 

While UBMS projects also provided services to students during the academic year in the 
mid-1990s, these services were minimal compared to UBMS summer services and also typically 
far less numerous and less intense than academic year services provided in regular Upward 
Bound. During the 1994-95 academic year, about one -third of UBMS projects provided tutoring 
or study sessions, and just over half provided assistance with college applications (see Exhibit 
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Exhibit 11.11 



Academic Year Services Offered by UBMS Projects, 1994-95 




0% 20% 40% 60% 80% 100% 

Percent of UBMS Projects 



Source: 1 998 survey of UBMS projects. 



II. 11). The average UBMS project provided about three types of these services during 1994- 
95. Not surprisingly, geographic proximity to their participating students influenced whether 
UBMS projects provided certain services during the academic year. UBMS projects were 
substantially more likely to provide tutoring and workshops during the academic year if a 
relatively large percentage of their participants lived in the same city or town as the program 
host. 25 



6. Instructional Approaches 

In 1994, the most common instructional approach taken by UBMS projects was the 
provision of instruction through courses in separate subjects. Four out of five UBMS projects 
offered courses in separate subjects (see Exhibit 11.12, Panel A). In three out of four UBMS 
projects, the primary method of instruction was either the provision of these courses (37 percent) 



24 By 1998-99, the percentage of UBMS projects regularly providing these services had increased substantially. 
For example, the percentage providing tutoring or study sessions rose from 81 percent to 93 percent, and the 
percentage providing assistance with college applications rose from 52 percent to 78 percent. This probably reflects 
the establishment of more locally oriented UBMS projects. 

25 For example, more than three-quarters of UBMS projects with a relatively large percentage of participants 
from the local area (above the median) offered tutoring during the academic year, versus less than half of UBMS 
projects with a relatively small percentage (below the median). 
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or the combination of these courses with interdisciplinary courses (also 37 percent, see Exhibit 
11.12, Panel B). 



However, UBMS projects frequently employed other instructional methods. The majority of 
projects (63 percent) offered interdisciplinary courses, and in a large minority (35 percent), at 
least some students worked on a large project or experiment that spanned multiple academic 
subject areas (see Exhibit 11.12, Panel A). 

UBMS projects vary considerably in how they sort students into classes or groups, in the 
summer of 1994, about half of projects placed their students in instructional groups based on 
proficiency level (37 percent) or grade level (16 percent). About one-fourth placed students with 
diverse proficiency levels in the same group to facilitate learning (presumably the learning of 
less proficient students), and the remaining projects grouped students by their interests or in 
some other way. 

UBMS participants do not spend most of their time in traditional lecture-style classes. At 
the average project during the summer of 1994, only one-fourth of the time was spent in lecture- 
style classes like those offered in most schools. The remaining time was spent in small group, 
teacher-led instruction (32 percent), laboratories (29 percent), computer-based instruction (12 
percent), and other settings (4 percent). 



D. INTENSITY AND QUANTITY OF SERVICES 

The services that UBMS projects offer and the length of their summer residential summer 
programs suggest that these projects offer intensive programs that provide students with a “large 
dose” of services, at least for one summer. Furthermore, larger doses of effective services may 
yield larger impacts than smaller doses, as we found for regular Upward Bound (Myers et al. 
2004). Summary measures of program intensity presented in this section indicate that UBMS 
projects offer an intensive program that might be expected to improve the math and science 
preparation of program participants. 

UBMS is a resource-intensive program. Program grants to UBMS projects provided an 
average of approximately $4,800 per student in FY 2004 (see Exhibit 11.13). This is comparable 
to funding for regular Upward Bound — approximately $4,500 per student — and much more 
expensive than most precollege programs. UBMS funding supports an extensive package of 
instruction and services, as described earlier in the chapter. 

Participants devote a substantial amount of time to the program, and most of this time is 
spent on academics. At the average UBMS project in the summer of 1994, students spent about 
29 hours per week receiving instruction and almost 1 1 hours per week on tutoring and 
homework. Thus, participating in UBMS over the summer is somewhat like having a full-time 
job requiring 40 hours per week. Because the vast majority of UBMS summer programs last six 
weeks, participants at the average project spend 240 total hours on academics during the 
summer. 



- Only five UBMS projects reported a program of a different length — three had a five-week session, one had a 
seven-week session, and one had an eight-week session. 
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Exhibit 11.12 



Instructional Methods Used by UBMS Projects, 1994 
A. Use of Different Instructional Methods 



Separate subjects 



Interdisciplinary courses 



Large project 



Other 

0% 20% 40% 60% 80% 100% 

Percent of UBMS Projects 



B. Primary Method Used 



Separate subjects 
and 
i 

courses 
37% 



Interdisciplinary 
courses 
9% 

Note: Percentages may not sum to 100 percent due to rounding. 

Source: 1 998 survey of UBMS projects. 
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Exhibit 11.13 



Per-Capita Funding for UBMS Projects, FY 2004 



Regular Upward 
Bound 

UBMS 

$0 $2,000 $4,000 $6,000 

Source: U.S. Department of Education Web page (www.ed.gov/programs/triomathsci/funding.html and 

www.ed.gov/programs/trioupbound/funding.html accessed June 2006). 




The amount of time devoted to core program activities varied across UBMS projects. For 
example, four projects reported that students spent over 40 hours per week in instruction alone, 
while a handful of projects reported estimates of less than 10 hours per week. Estimates of time 
spent on tutoring or homework ranged from 0 to 20 hours per week. And the estimates of 
average total time spent on academics during the summer varied considerably, from under 100 
hours at some projects to over 340 hours at some others. 

Most UBMS participants seem to stay for the entire summer. At the average project in the 
summer of 1994, 94 percent of participants completed all the requirements of the program. A 
few projects reported completion rates of 50 percent or less, but this does not necessarily indicate 
a high dropout rate. Students might have attended the summer program for its full length but be 
counted as failing to “complete all the requirements of the program” — the wording in the 
questionnaire — because they did not, for example, turn in all their assignments. 

Some students participate for multiple summers or receive services during the academic 
year. As indicated earlier, about 70 percent of UBMS projects operating in summer 1994 
allowed students to return from the previous summer. Among those projects, an average of 35 
percent of 1994 participants had also participated during 1993. Furthermore, some UBMS 
projects extended students’ exposure to the program beyond the summer by providing services 
during the academic year (see Section C.4 for more details). 



21 In summer 1998, not only did more UBMS projects (over 90 percent) allow students to return, but at those 
projects an average of 52 percent of students had also participated in summer 1997. 
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III. THE IMPACTS OF THE UPWARD BOUND MATH-SCIENCE PROGRAM 



UBMS was established to increase economically disadvantaged students’ achievement in 
high school math and science courses, to increase the likelihood that they would major in math 
and science in college, and ultimately to increase their representation in math and science 
careers. Until now, no rigorous studies have measured the extent to which the program achieves 
its goals. In this chapter, we assess the effects of UBMS on important outcomes for program 
participants four to eight years after graduating from high school. 

This chapter presents our estimates of the effects of UBMS on (1) high school preparation 
for college and for majoring in math and science; (2) college enrollment, highest level of college 
attended, college selectivity, years of college completed and highest degree earned; and (3) field 
of study in college. These estimates are based on infonnation collected in 2002 or earlier, and 
another round of data collection is currently underway. Given many students in our sample may 
have, for example, selected a major or completed a degree between 2002 and 2004, the findings 
in this report should be treated as an initial assessment of the effects of the program. Our final 
assessment will be based on the data collected in 2004. 

The evidence suggests that in addition to improving students’ grades in high school, UBMS 
increases the likelihood of: 



• Taking chemistry and physics in high school 

• Enrolling in more selective four-year colleges and universities 

• Completing a postsecondary degree in a math and science field 



The effects of UBMS are nontrivial and in the expected direction. For example, UBMS 
increased the likelihood of earning a bachelor’s degree in a math or science field four to eight 
years after high school graduation from 12 percent to 20 percent, in the next report on UBMS, 
we will weigh the size of the program’s benefits against its costs. 

Because UBMS participants in our sample were not randomly selected to participate in the 
program, the impact estimates presented in this chapter may suffer from selection bias. 
However, in designing the study, we identified several likely sources of bias and have addressed 
them through a combination of data collection and statistical methods described later in the 
chapter. Therefore, the research design is stronger than in the typical nonexperimental study, 
and as a result, the impact estimates should be more credible. 



A. STUDY DESIGN 

UBMS provides intensive academic enrichment in math and science. Like regular Upward 
Bound projects, most UBMS projects also offer some assistance in preparing for college, such as 
assistance with college applications (see Chapter II for more details). The combination of 
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intensive academic enrichment in math and science with college preparation assistance suggests 
that UBMS might have positive effects on the following outcomes for program participants: (1) 
performance in high school, especially in math and science courses; (2) postsecondary 
attendance, persistence, and completion; and (3) the likelihood of completing a postsecondary 
degree in a math and science field. Therefore, we designed the analysis to answer the following 
three research questions: 



1 . What are the effects of UBMS participation on student performance in high school 
overall and in math and science courses in particular? 

2. What are the effects of UBMS participation on college attendance, attendance at 
different types of colleges and universities, years of college, and college completion? 

3. What are the effects of UBMS participation on the likelihood of completing a degree 
in math or science? 

We have designed the analysis to measure the impacts of UBMS on two important 
subgroups: students who had previously participated in regular Upward Bound and students who 
had not participated. UBMS participants who previously participated in regular Upward Bound 
may have received a large dose of precollege services and academic preparation before 
participating in UBMS. However, most of the other UBMS participants entered UBMS without 
having received such intensive services. It is reasonable to expect UBMS to have larger effects 
on the students who had not previously received intensive services: students who have already 
received them may have already received the boost they needed to succeed. On the other hand, 
UBMS participants may be better prepared to benefit from their participation if they have 
previously participated in intensive services, in our analysis, we compute the effects separately 
for students who had previously participated in regular Upward Bound and students who had not 
participated to assess whether the effects of UBMS depend on the amount of precollege services 
students have received to that point. Regular Upward Bound is just one of the other programs in 
which UBMS participants could have participated. However, in our analysis, regular Upward 

Bound deserves special consideration because many UBMS participants participate in regular 

28 

Upward Bound, and because few other programs are as intensive as UBMS. 

It is important to clarify we have not attempted to measure the effects of UBMS versus no 
precollege services. The analysis was designed to measure the effects of participating in UBMS 
relative to what students would have participated in otherwise — which might include regular 
Upward Bound — not the effects of participating in UBMS relative to no program participation. 
Most UBMS participants in our sample did not participate in regular Upward Bound, and only a 
few other precollege programs are as intensive as Upward Bound. Therefore, most UBMS 
participants would have participated in less intensive precollege services if they had not 
participated in UBMS. 



28 Moore (1997), Appendix A includes a list of 28 alternative math and science precollege programs. 
However, some other precollege programs without a specific precollege focus, such as regular Upward Bound, may 
share some common features with UBMS. Annual per student costs were available for 17 of the 28 programs; of 
these 17, only two were more expensive per student-year than UBMS. 



24 




In this section, we describe the design of the analysis used to measure the impacts of 
UBMS. The impact analysis is based on a matched comparison group that attempts to reduce 
two types of bias common to many nonexperimental studies: selection bias and bias attributable 
to different data collection protocols for the participant and comparison groups. The strength of 
the impact analysis rests on three features of its design: 



1 . How we selected our initial samples, particularly the comparison sample 

2. How we collected “baseline” (preprogram) infonnation on the two samples 

3. How we used that information to select a matched comparison group 

In the remainder of this section, we describe these three features of the study and our 
approach to estimating the effects of UBMS on student participants. 



1. Selecting the Samples 

For the impact analysis, we obtained our sample of program participants from the projects 
themselves. In 1998, we contacted the 62 Upward Bound Math and Science Centers (MSCs) 
that were operating at that time and that had been operating between 1993 and 1995.“ From 
these MSCs, we requested lists of the students who had participated in their program in the 
summer of 1993, the summer of 1994 or the summer of 1995. 30 To reduce the costs of collecting 
the necessary data, we selected one out of every four of the students from these lists for our 
analysis sample. 

A primary feature of any nonexperimental evaluation is the choice of a comparison group. 
Experiments yield the best comparison groups because differences in outcomes between treated 
and untreated cases cannot be attributed to selection bias. In the absence of an experiment, the 
strength of an evaluation depends on the comparability between the participant and comparison 
groups. 

The most convenient comparison group for the impact analysis is also a compelling one — 
and the one we used. For the comparison group, we selected students from the evaluation of 
regular Upward Bound who reported that they had not participated in an MSC. In doing, so, 
we selected a comparison group with three desirable attributes: 



29 While we would have been interested in obtaining lists of students from projects that were no longer 
operating in 1998, we believed that it would be very difficult to obtain such lists. If MSCs that closed before 1998 
operated less effective programs than those that remained open, then the results presented in this chapter may 
overstate the effectiveness of MSCs operating between 1993 and 1995. 

30 We received participant lists from all but one of these MSCs. 

31 If UBMS projects admitted eligible applicants on a fairly random basis, rejected applicants would probably 
constitute the best comparison group. However, MSCs are not required to select randomly from eligible applicants, 
and the evidence suggests that they do not. Case studies of 14 MSCs in the mid-1990s suggest that many MSCs 
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1. Like UBMS participants, comparison students applied to participate in Upward 
Bound. Therefore, both UBMS participants and comparison students revealed a 
high level of motivation to pursue precollege services. This provides some 
protection against a common source of selection bias in nonexperimental studies — 
bias from comparing more motivated participants to less motivated nonparticipants. 

2. Like UBMS participants, comparison students met the federal eligibility 
requirements to participate in some type of Upward Bound program — regular or 
math and science. The federal eligibility requirements are the same for both regular 
Upward Bound and UBMS. To be included in the sample for the evaluation of 
regular Upward Bound — the sample from which we selected our comparison 
group — a student must have applied to a regular Upward Bound project and been 
determined eligible to participate. Therefore, both UBMS participants and 
comparison students in our sample must either have come from “low-income” 
families (income below 150 percent of the poverty line) or potential “first- 
generation” families (neither parent had earned a bachelor’s degree). 

3. Like UBMS participants, many comparison students would have met project- 
specific eligibility requirements imposed by some MSCs. Moore (1997) indicates 
that MSCs often apply additional admissions criteria in selecting applicants — criteria 
that include a minimum grade point average (GPA) in math and science, in this 
chapter, we show that many comparison students met the same criteria: many were 
successfully “matched” to UBMS participants who took similar courses and earned 
similar grades in math and science in ninth grade. 



While the regular Upward Bound sample is a useful comparison group for measuring the 
effects of UBMS, it is not a perfect one. Data from the 1990s suggest that MSCs typically had 
more stringent minimum GPA requirements than regular Upward Bound projects; and MSC 
staff — who typically had recent experience with regular Upward Bound participants — reported 
that UBMS participants tended to be “more serious about school” than regular Upward Bound 
participants (Moore 1997, p. 26). Therefore, UBMS participants might fare better than regular 
Upward Bound participants even without any assistance from UBMS, and simple differences in 
mean outcomes between the UBMS participant and comparison groups may overstate the effects 
of UBMS due to selection bias. 

To reduce selection bias, we selected a matched comparison sample from the regular 
Upward Bound sample. More specifically, we matched each UBMS participant to one or more 



(continued) 

impose additional eligibility requirements beyond the federal requirements, including an interest in math and science 
(Moore 1997). Therefore, it is unlikely that rejected applicants would have the virtues of a randomly selected 
control group for the purposes of this evaluation. Furthermore, the difficulty in obtaining lists of program 
participants from MSCs several years after they participated in UBMS suggests that many MSCs would not have 
been able to provide information on rejected eligible applicants. 
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regular Upward Bound sample members with similar characteristics based on data collected from 
student surveys and transcripts. 32 See Section 3 for more details. 



2. Collecting Baseline Data 

The baseline variables constructed for the impact analysis characterize members of the two 
samples early in high school before UBMS participants in our sample entered UBMS. These 
variables are critical because they allow us to account — through a combination of matching and 
regression adjustments — for many preexisting differences between the two groups that might 
otherwise bias our impact estimates. To collect the information necessary to create baseline 
variables, we conducted student surveys and collected high school transcripts (see Appendix A). 

An important strength of the study’s design is that the data collection strategy was similar 
for both samples. Heckman et. al. (1997) argue that different survey questionnaires for the 
participant and comparison samples can be an important source of bias in nonexperimental 
studies. In this evaluation, we developed the initial survey questionnaire for UBMS participants 
from the baseline survey questionnaire for the regular Upward Bound sample. Therefore, the 
survey questions from which we constructed baseline variables were often identical and always 
similar for the two samples. Furthermore, the approach to collecting and coding high school 
transcripts was the same for the two samples. Therefore, it is unlikely that differences in the data 
between the two samples have biased the impact estimates presented in this chapter. 

The baseline variables for the impact analysis fall into the following three categories: (1) 
demographic and family characteristics, (2) participation in other precollege programs and (3) 
ninth-grade academic achievement in math and sciences and more generally (see Exhibit III. 1). 
We believe that the measures of students’ ninth-grade academic achievement are critical to the 
strength of the study. Given the findings in earlier reports, it seems entirely possible that even 
among students with similar demographic and family characteristics, students who participated 
in UBMS might have a higher academic aptitude and interest in math and science than students 
who participated in regular Upward Bound. Therefore, we use information on ninth-grade 
courses taken and ninth-grade grade point average — overall and specifically math and science — 
to control for differences between the two samples in early high school achievement. 



32 Very few of students in the regular Upward Bound sample participated in UBMS, and those who reported 
participating in UBMS were excluded from the comparison group. 

33 Two differences in the two surveys were addressed in constructing baseline variables for the MSC impact 
analysis: (1) the two survey questionnaires were different, and (2) the surveys were conducted at different times — 
after high school for the participant group and early in high school for the comparison group. To address the 
differences in the survey questionnaires, we used data from survey questions that are either identical or almost 
identical. To address the difference in the timing of the surveys, we used data from survey questions only when the 
timing of the survey was unlikely to affect the answer to the question. 
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Exhibit III. 1 



Baseline Variables 


Category 


Variables 


Source 


Demographic and family 


Sex 

Race and ethnicity 
Census region 
Native English speaker 
Mother’s time in the U.S. 

Sibling participated in Upward Bound 
High school cohort 


Initial surveys 


Prior program participation 
Ninth-grade achievement 


Sample member participated in regular Upward Bound 

GPA in math and science 
GPA in other subjects 
Math course taken 
Science course taken 


Initial survey and project records 
High school transcripts 


Note: The initial surveys 


were conducted in 1999 for UBMS participants and 


in 1992-94 for comparison 



students in the regular Upward Bound sample. To identify UBMS participants who had previously 
participated in regular Upward Bound, we used responses to the 1999 initial survey; to identify 
comparison students who participated in regular Upward Bound, we used participation information 
provided by projects. We collected high school transcripts in 2000 and 2003 for UBMS participants and 
in 2000 and earlier years for comparison students. 



3. Selecting a Matched Comparison Sample for the Impact Analysis 

The data collected on UBMS participants and regular Upward Bound sample members are 
useful in identifying many similarities between the two groups — and some differences as well 
(see Exhibit III.2). In both groups, 37 percent of the students were African American and 1 1 
percent had a sibling who had participated in Upward Bound. The two groups were similar with 
respect to many other characteristics as well, including the percentage taking algebra in ninth 
grade. However, the two groups exhibit differences on several dimensions. For example, 
UBMS participants were more likely to be male and tended to have higher grades than regular 
Upward Bound sample members. 

To address possible selection bias, we selected a matched comparison group from the 
regular Upward Bound sample using propensity score matching methods. 34 The goal in 



34 Many studies adjust for baseline differences of these types using standard covariance adjustments — that is, 
by controlling for these variables in a regression analysis. However, the differences between the UBMS participant 
sample and the regular Upward Bound sample are too large to expect covariance adjustment to be reliable. 
Regression adjustments are likely to be unreliable if the means of the propensity scores are more than half a standard 
deviation apart (Rubin 2002). For the participant sample, the mean and standard deviation of the propensity score 
are 0.58 and 0.22, respectively; for the comparison sample, the mean and standard deviation of the propensity score 
are 0.25 and 0.23, respectively. Therefore, the difference in mean propensity scores is more than one standard 
deviation, and regression adjustments alone are likely to be unreliable. 
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Exhibit III. 2 



Summary Statistics from the Baseline Variables 
(Percentage unless otherwise noted) 



Characteristic 


UBMS 

Participants 


Regular Upward 
Bound Sample 


Matched Comparison 
Group 


Participated in Regular Upward Bound 


18 


55*** 


18 


Female 


59 


72*** 


59 


Race and Ethnicity 


African American 


37 


37 


37 


White 


25 


24*** 


30 


Hispanic 


18 


20 


16 


Other race 


20 


9 *** 


17 


Region 


Northeast 


11 


9 


12 


Midwest 


23 




28 


South 


40 


45** 


35 


West 


25 


26 


24 


Entry to High School 


1991-92 


32 


25 *** 


28 


1992-93 


39 


49 *** 


37 


1993-94 


29 


26 


35 


Other Characteristics 


Native English speaker 


80 


gy*** 


86 ** 


Mother in U.S. most of her life 


79 


£7*** 


83* 


Siblings in Upward Bound 


11 


11 


12 


Ninth-Grade Math Course 


Lower than algebra 


16 


32*** 


14 


Algebra 


55 


53 


59 


More than algebra 


29 


14*** 


27 


Ninth-Grade Science Course 


Biology, chemistry, or physics 


37 


26*** 


37 


Ninth-Grade GPA (mean) 


Math and science 


2.69 


1.65*** 


2.71 


Other subjects 


3.24 


2.64*** 


3.25 



*/**/*** Significantly different from UBMS participants at the 0.10 / 0.05 / 0.01 level. 
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matching was to select a matched comparison sample from the regular Upward Bound sample 
such that the distributions of the baseline variables for the UBMS participant sample and the 

or 

matched comparison sample were similar. Matching was conducted separately for sample 
members who had previously participated in regular Upward Bound and for those that had not: 



• UBMS participants who had previously participated in regular Upward Bound were 
matched to members of the treatment group in the evaluation of regular Upward 
Bound. 

• UBMS participants who had not previously participated in regular Upward Bound 
were matched to regular Upward Bound control group members who did not 
participate in regular Upward Bound. 



Regular Upward Bound sample members that were matched to at least one UBMS participant 

36 37 

were included in the matched comparison sample. ’ 

Using matching procedures, we were able to select a matched comparison group that is 
highly similar to the sample of UBMS participants on many dimensions (see the last column of 
Exhibit III. 2). Only two of the differences between the groups are statistically significant — the 
difference in the percentage of sample members who are native English speakers and the 
percentage whose mother has lived in the United States for all or almost all of her life. 
Furthermore, given the number of baseline variables, two is a small number of significant 
differences: we would expect about two significant differences even if the differences between 
the two groups were purely random. While there is no guarantee that matching removed all 
unmeasured differences between the two samples, matching removed differences on a broad 
range of baseline variables — differences that might otherwise bias the impact estimates. 



35 More specifically, the goal is to ensure that the distributions of the baseline variables for the UBMS 
participant sample and the matched comparison sample are similar enough such that covariance adjustments will 
produce reliable impact estimates 

36 The propensity score matching and the impact analysis are restricted to sample members who entered high 
school between 1991 and 1993. While some UBMS participant sample members entered high school before 1991 
and after 1993, relatively few comparison sample members did so. Furthermore, high school cohort is related to the 
likelihood of participating in the program and to the outcomes of interest because earlier cohorts had more time to 
enter college and select a field of study by the time they were interviewed for the evaluation. Therefore, to protect 
the internal validity of the study, we focused the analysis on students who entered high school between 1991 and 
1993. 



37 UBMS participants could be matched to more than one regular Upward Bound sample member, and regular 
Upward Bound sample members could be matched to more than one UBMS participant. To be matched, a pair of 
students must satisfy the following condition: the difference between matched students in the log odds of the 
propensity scores was less than 0.20 times the standard deviation of the log odds. Smith and Todd (2003) refer to 
this type of matching as “radius matching.” 
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4. Estimating the Impacts of UBMS Participation 

To measure the effects of UBMS participation on participating students, we used a 
regression-based approach that allows us to (1) adjust for the small remaining differences 
between the UBMS participant sample and the matched comparison group and (2) increase the 
precision of our impact estimates. The regression models yield estimates of the effect of UBMS 
on students who participated in the program. 38 

We estimated the effects of UBMS for the entire sample and separately for selected 
subgroups of students. The effects of UBMS may depend on the amount of other precollege 
services received. Therefore, we present separate impact estimates for those who participated in 
regular Upward Bound and those who did not participate. Furthermore, we present separate 
estimates by sex and race since achievement in math and science often varies along these 
dimensions, and the effects of UBMS may vary along the same dimensions. 39,40 



B. ANALYSIS AND FINDINGS 

In this section, we present the estimated effects of UBMS on students’ outcomes. 
Consistent with program objectives, participation in UBMS seems to promote postsecondary 
study in math and science. Specifically, our estimates suggest that UBMS participation (1) raises 
grades in high school and course-taking in chemistry and physics, (2) increases enrollment in 
more selective four-year institutions and the number of years of college completed, and (3) raises 
the likelihood of both pursuing postsecondary studies and completing a four-year degree in math 
or science. In contrast, UBMS does not affect course-taking in advanced math in high school. 



38 We regressed each outcome on a set of control variables and an indicator of whether the student participated 
in UBMS. The control variables included the variables used in selecting the matched comparison group: prior 
participation in regular Upward Bound, siblings in Upward Bound, sex, race, ethnicity, mother’s native language 
and immigrant status, high school cohort, region of the country, and several variables describing academic 
achievement in ninth grade, including GPA — separately for math and science courses and for other courses — math 
course taken, and science course taken. For continuous variables, such as number of college credits, we estimated 
linear regression models; for categorical outcomes, such as whether the sample member pursued postsecondary 
studies in math or science, we estimated logistic regression models or “logit” models. In estimating standard errors, 
we accounted for clustering by project and used the Taylor series linearization methods employed by the SUDAAN 
statistical analysis software. 

39 Other reports from the National Evaluation of Upward Bound have shown that the effects of regular Upward 
Bound vary with the educational expectations of student applicants (see Myers et al. 2004 and Myers and Schirm 
1999). In particular, regular Upward Bound greatly increased the likelihood of attending a four-year college or 
university for students who did not expect to earn a bachelor’s degree when they applied to participate in regular 
Upward Bound. Because we first interviewed UBMS participants after they finished participating in UBMS, we 
lack information on their educational expectations when they applied to participate in UBMS. Therefore, we cannot 
assess whether the effects of UBMS vary with students’ educational expectations before participating. 

40 In the national evaluation, we have estimated the effects of regular Upward Bound separately by eligibility 
category (low-income, first-generation, or both). Flowever, student data was provided by project directors when 
they submitted students to Mathematica for random assignment. Because we did not conduct random assignment 
for the evaluation of UBMS, we lack data on eligibility for UBMS participants. While we did obtain rosters of prior 
participants from UBMS projects in 1998, we did not request data on eligibility because we did not believe that 
projects could easily provide them for students who participated three or more years ago. 
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1. The Effect of UBMS on High School Outcomes 



The UBMS program is designed to strengthen participants’ math and science skills. MPR 
conducted follow-up student surveys and collected high school transcripts to determine whether 
UBMS participation increased the amount of course work in math and science as well as total 
high school credits, high school graduation and GPA in math and science and overall. 41 

UBMS raised the likelihood of taking upper-level high school courses in science but not in 
math (see Exhibit III. 3). Our estimates indicate that UBMS increased the likelihood of taking 
upper-level science courses in high school, raising the percentage of students taking chemistry 
from 78 to 88 percent and raising the percentage of students taking physics from 43 to 58 
percent. 42 The estimated effects of UBMS on the likelihood of taking algebra II, trigonometry, 
precalculus and calculus are statistically insignificant. 

It is not clear why UBMS affected course-taking in the sciences but not in math. One 
possibility involves differences between the type of instruction that MSCs offer in math and the 
type of instruction they offer in sciences. Almost two-thirds of MSCs reported offering 
interdisciplinary classes, and about a third of them offered instruction that revolved around a 
large project or experiment (see Chapter II). However, findings from Moore (1997) suggest that 
in MSCs, laboratories and hands-on instruction were much more common in science instruction 
than in math instruction, and that MSC instructors used math to help student prepare for careers 
in science and technology (see Moore 1997, pp. 41-42). It is possible that the combination of 
hands-on instruction offered only in science with the science orientation of MSC instructors is 
responsible for UBMS’s effects on science course-taking. 

Additional estimates suggest UBMS raises overall achievement and educational attainment 
in high school (see Appendix B). UBMS had a positive effect on high school grades, increasing 
the average GPA in math courses from 2.7 to 2.8, the average GPA in science courses from 2.7 
to 2.9 and the average GPA overall. It also raised the graduation rate from 96 to 99 percent. 

UBMS appears to have had larger effects on grades and science course work for Hispanics 
than for African Americans (see Appendix B). For example, UBMS raised the average GPA in 
math courses from 2.4 to 2.5 for African Americans and from 2.4 to 2.7 for Hispanics. In 
addition, UBMS raised the likelihood of taking chemistry and physics by 7 percentage points 
each for African Americans and by 17 and 27 percentage points, respectively, for Hispanics. 43 
Comparisons of males to females and students who had participated in regular Upward Bound to 
other students revealed few significant differences. 



41 With the exception of the outcome variables describing high school completion, which were constructed 
from student survey responses, the high school outcomes were constructed using data from high school transcripts. 

42 These increases in part reflect increases in the total number of courses taken in high school: UBMS raised 
the average number of high school credits earned from 25 to 26. 

43 For African Americans, the estimated effect on the likelihood of taking physics was not statistically 
significant. 



32 




Exhibit III. 3 



Math and Science Courses Taken in High School 
(Percentage of students) 





UBMS Participants 


Matched Comparison Group 


Impact 


Math Courses 


Algebra 


97 


97 


0 


Geometry 


86 


86 


0 


Algebra II 


73 


74 


-1 


Trigonometry 


35 


30 


5 


Analysis or Precalculus 


36 


31 


5 


Calculus 


26 


24 


2 


Science Courses 


Biology 


97 


98 


-1 


Chemistry 


88 


78 




Physics 


58 


43 


15*** 



*/**/*** Statistically significant at the 0.1 / 0.05 / 0.01 level. 

NOTES: Impact may not exactly equal the difference between UBMS participants and the matched comparison 

group due to rounding. See Table B.l in Appendix B for more details and additional high school 
outcomes and Tables B.2 through B.4 for estimates for subgroups of UBMS participants. 



2. The Effect of UBMS on Postsecondary Attendance, Persistence, and Completion 

While UBMS focuses on preparing students to major in math and science and to complete a 
degree in a math and science field, a person must enroll in college before choosing a major and 
must complete college to earn a degree in a math and science field. Even if UBMS had little 
effect on students’ choice of major, UBMS might be a cost-effective strategy to increase college 
enrollment and completion for disadvantaged students. Therefore, we assess whether UBMS 
promotes postsecondary attendance, persistence, and completion before examining its effects on 
college major. 

The evidence suggests that UBMS participation increases the likelihood of attending a 
postsecondary institution. Estimates for the matched comparison group suggest that 90 percent 
of UBMS participants would have attended a postsecondary institution if they had not 
participated in UBMS (see Exhibit III.4). This indicates that UBMS participants are much more 
likely to attend college than the average low-income high school student, and a 90-percent 
attendance rate leaves little room for improvement. Despite this, the evidence suggests that 
UBMS participation increases the likelihood of attending a postsecondary institution from 90 to 
95 percent, although as shown in Appendix C, this result is sensitive to how postsecondary 
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Exhibit 111.4 



Postsecondary Attendance, Highest Level Attended and College Selectivity 
(Percentage of students) 





UBMS Participants 


Matched Comparison Group 


Impact 


Postsecondary Attendance 


95 


90 




Highest Level Attended 


Four-year college or university 


82 


71 


1 J*** 


Two-year college 


11 


16 




Vocational institution 


2 


4 


-2 


Most Selective Four-Year institution 


More selective 


33 


23 


ii 


Less selective 


48 


48 


0 



*/**/*** Statistically significant at the 0.1 / 0.05 / 0.01 level. 

NOTES: Impact may not exactly equal the difference between UBMS Participants and the Matched Comparison 

Group due to rounding. “Most Selective College or University” refers to four-year colleges and 
universities only, and some four-year institutions were not classified in Barron ’s Profiles of American 
Colleges (2003). See Table B.5 in Appendix B for more details and additional postsecondary outcomes 
and Tables B.6 through B.8 for estimates for subgroups of UBMS participants. 



attendance is measured and, specifically, to whether college transcripts are used to verify 
attendance. 44 

The evidence also suggests that UBMS participation increases the likelihood of attending a 
four-year college or university from 71 percent to 82 percent. The increase in four-year 
attendance can be attributed to increased attendance at more selective schools as UBMS raises 
the likelihood of attending such a school from 23 to 33 percent 45 It appears that some of this 



44 This 95-percent figure suggests that almost all UBMS participants attend a postsecondary institution in the 
first few years after high school. In contrast, findings from a recent analysis of performance data submitted by 
UBMS projects seem to suggest a much lower rate of postsecondary attendance. Analyzing the performance data, 
Curtin and Cahalan (2004) find that only 68 percent of UBMS participants who graduated from high school in 1999- 
2000 were reported as attending a postsecondary institution in 2000-01. The difference between this figure and the 
enrollment rate presented in this report reflects substantial differences in how the two estimates were derived. 
According to information provided to us by ED, the estimate in Curtin and Cahalan (2004) indicates the percentage 
of former program participants for whom UBMS project directors could confirm postsecondary attendance. The 
estimate was not intended to measure the postsecondary enrollment rate per se, and it understates the tine rate for 
UBMS participants for two related reasons. First, the estimate includes in the denominator but not the numerator 
UBMS participants from projects that were no longer funded and, therefore, did not submit performance data. 
Second, it excludes from the numerator about one-fifth of the participants who are included in the denominator 
because UBMS project staff were unable to confirm college attendance for these former participants. For our 
estimate, in contrast, we exclude from both the numerator and denominator the UBMS participants in our sample for 
whom we are missing data due to survey nonresponse and, then, use standard weighting techniques to adjust for the 
missing observations. 

45 If a school was rated as “most competitive,” “highly competitive,” or “very competitive,” then we classified 
the school as “more selective.” If a school was rated as “competitive,” “less competitive,” “noncompetitive,” or 
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increase in four-year college enrollment can be attributed to students who would otherwise have 
attended only two-year colleges: UBMS reduced the likelihood of enrolling in a two-year 

college but not in a four-year college or university from 16 to 11 percent. 

UBMS increased the number of years that students spend in four-year colleges and 
universities without increasing the completion rate at these schools. Our estimates suggest 
UBMS increased the number of years enrolled in a four-year institution from 2.4 to 2.9 (see 
Exhibit III. 5). While UBMS increased the amount of time participants spent in college, it had no 
effect on the likelihood of their completing a bachelor’s degree as of the time we last interviewed 
them in 2002. Given that almost half of UBMS participants in our sample (47 percent) were 
attending four-year colleges and universities when we last interviewed them, it is too early to 
reach firm conclusions about the effects of UBMS on the number of years of college attended 
and on college completion. Our final report on the Upward Bound Math-Science Program will 
be based on data collected in 2004. 

The average effects of UBMS mask some interesting differences between men and women, 
for both men and women, UBMS increased the percentage attending a selective four-year 
institution — from 23 to 34 percent for men and from 21 to 32 percent for women. However, 
UBMS only affected overall four-year enrollment for women. For women, UBMS increased the 
likelihood of attending a four-year institution from 68 to 82 percent and reduced the likelihood of 
attending a two-year institution (without also attending a four-year institution) from 1 8 percent to 
1 0 percent; for men, both of these effects were statistically insignificant. 

The effects on college completion also differ between men and women. For women, the 
positive effect of UBMS on enrollment in four-year institutions — along with positive effects on 
credits earned in these institutions — translates into higher graduation rates from four-year 
institutions. UBMS raised the percentage of women earning a bachelor’s degree from 32 to 40 
percent. For men, the effect of UBMS on the percentage completing a bachelor’s degree is 
statistically insignificant; however, UBMS increased the percentage completing an associate’s 
degree from 4 to 8 percent. 

In the last report from the national evaluation (Myers et al. 2004), we presented two sets of 
estimates for postsecondary attendance and highest level attended — the first based entirely on the 
information reported by the student, and the second that required verification of enrollment from 
the institution. In computing the second set of estimates, we made the strong assumption that a 
student did not attend the school they reported to us unless we received verification from the 
school in the form of a college transcript (or a reason for not providing one that clearly indicated 
that the student had attended the school). However, additional analysis of the data suggests that 
in most cases for which we did not receive a transcript, there is no reason to doubt that the 
students actually attended the schools they reported. Schools provided a variety of reasons for 



(continued) 

“special,” or the school was either excluded from Barron’s or was included but not rated, then we classified the 
school as “less selective.” Students were then classified as “more selective” school if they attended one or more of 
the “more selective” institutions, and “less selective” if they attended one or more of the “less selective” institutions 
and did not attend a “more selective” institution. 
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Exhibit III. 5 



Years of College and Degree Completion 





UBMS Participants 


Matched Comparison Group 


Impact 


Years of College 


Four-year college or university 


2.9 


2.4 


Q 4** 


Two-year college 


0.4 


0.4 


0.0 


Degree Completion (percent) 


Bachelor’s degree 


35 


33 


2 


Associate’s degree 


7 


7 


-1 



*/**/*** Statistically significant at the 0.1 / 0.05 / 0.01 level. 

NOTES: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section 

A. 4). Impact may not exactly equal the difference between UBMS Participants and the Matched 
Comparison Group due to rounding. Y ears of Postsecondary Education were computed by dividing the 
number of credits earned by 30. See Table B.5 in Appendix B for more details and additional 
postsecondary outcomes and Tables B.6 through B.8 for estimates for different subgroups of UBMS 
participants. 



not providing transcripts, including confidentiality considerations and money that the student 
owed to the school. Therefore, for all outcome variables except years of college, which are 
based on information from college transcripts, the estimates presented in this chapter are based 
on the information reported by the students themselves. However, we conducted a sensitivity 
analysis to see whether requiring verification of enrollment affected our impact estimates, and in 
most cases, it did not. See Appendix C for more details about the sensitivity analysis. 



3. The Effect of UBMS on Postsecondary Field of Study 

The primary objective of UBMS is to prepare students for postsecondary studies in math and 
science. Using information reported by students in 2002, we conducted an analysis to examine 
whether UBMS participation increases the likelihood of pursuing postsecondary studies in math 
and science or the likelihood of earning a postsecondary degree in a math or science fields. To 
determine whether a sample member had pursued postsecondary studies in math and science, we 
asked sample members in the 2002 survey for their “most recent or intended field of study,” and 
we classified their responses according to the same classification system used in the National 
Science Foundation’s Scientists and Engineers Statistical Data System (SESTAT). 46 
Furthermore, for the analysis, we separated the social sciences from other math and science 
fields (which we refer to as “math or science”) because the objectives of the program are more 
closely tied to the latter than the fonner. 

46 The fields classified as science and engineering were biological sciences, computer science, engineering, 
mathematics, physical sciences, and technical fields. The fields classified as non-science and engineering were 
agriculture, arts, business, education, clerical or legal assistance, communications, health-related fields, humanities, 
trade and industry, protective services and consumer or personal services. A small number of fields reported by 
sample members could not be classified as either science and engineering or nonscience and engineering. (For more 
details about NSF’s classification of fields of study, see www.nsf.gov/sbe/srs/nsf99337/pdf/appa.pdf.) 
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The evidence suggests that UBMS participation encourages students to pursue 
postsecondary studies in math or science and also in the social sciences. UBMS increased the 
likelihood of majoring or intending to major in math or science from 23 to 33 percent overall and 
from 18 to 28 percent if we focus on majors at four-year colleges and universities (see Exhibit 
III.6). 47 The effects are also positive but smaller in magnitude for the social sciences: UBMS 
increased the likelihood of pursuing postsecondary studies in the social sciences from 7 to 1 1 
percent overall and from 7 to 10 percent if we focus on majors at four-year institutions. 

The evidence also suggests that UBMS participation may encourage students to complete 
postsecondary degrees in math or science. However, with almost half of participants still in 
college, it is too early to reach final conclusions about UBMS’s effects on college completion 
overall or in specific fields. As of the 2002 survey, the effect of UBMS on the likelihood of 
earning a degree or certificate in math and science was statistically insignificant. However, if we 
focus on four-year institutions, UBMS altered the types of fields in which students earned 
degrees. More specifically, it raised the likelihood of earning a bachelor’s degree in math or 
science from 6 to 12 percent and reduced the likelihood of earning a bachelor’s degree outside of 
math or science or the social sciences from 20 to 14 percent. 

The effect of UBMS on the likelihood of majoring in math and science was larger for men 
than for women. While UBMS increased the likelihood of pursuing postsecondary studies in 
math and science from 38 to 58 percent for men, the estimated effect for women is much smaller 
and statistically insignificant. In contrast, there is no evidence that the effect of UBMS on the 
likelihood of completing a math or science degree differs between the two groups. 



C. INTERPRETATION OF THE FINDINGS 

The estimated effects of UBMS paint a fairly consistent picture of UBMS helping its 
participants onto an educational path that could lead to careers in math and science. Our findings 
suggest that UBMS participation improves student outcomes in high school and college, and — 
consistent with the objectives of the program — increases the odds of completing a college degree 
in math and science. 

While we took several steps to reduce selection bias, it is certainly possible that the true 
effects of participating in UBMS are smaller than our estimates suggest. As reported earlier in 
the chapter, UBMS participants may be more serious about school than regular Upward Bound 
participants on average. This difference is reflected in our data: UBMS participants in our 
sample had higher GPAs and took more advanced math and science courses in ninth grade than 
members of the regular Upward Bound sample. We accounted for these differences by selecting 
a matched comparison group that resembled the UBMS participant sample in both grades earned 
and course-taking in ninth grade. However, it is possible that despite earning similar grades and 
taking similar courses early in high school, the UBMS participant sample is somewhat more 
serious about school, more serious about math and science, or is different from the matched 
comparison group in some other way that would lead our analysis to overstate the effects of 
UBMS. 



47 For convenience, we use the terms “major” and “field of study” interchangeably. 
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Exhibit III. 6 



Field of Study 
(Percentage of students) 





UBMS Participants 


Matched Comparison Group 


Impact 


All Postsecondary institutions 

Postsecondary Studies 


Math or science 


33 


23 




Social science 


11 


7 


4* 


Other 


42 


51 


_^*** 


Postsecondary Studies Completed 


Math or science 


15 


12 


3 


Social science 


8 


4 


3* 


Other 


21 


28 


_y** 


Four-Year Colleges and Universities 

Postsecondary Studies 


Math or science 


28 


18 


io*** 


Social science 


10 


7 


4 * 


Other 


36 


39 


-3 


Postsecondary Studies Completed 


Math or science 


12 


6 


g*** 


Social science 


6 


4 


3 


Other 


14 


20 


- 6 ** 



*/**/*** Statistically significant at the 0.1 / 0.05 / 0.01 level. 

NOTES: Impact may not exactly equal the difference between UBMS Participants and the Matched Comparison 
Group due to rounding. See Table B.9 in Appendix B for more details and additional outcomes and Tables 
B. 10 through B.12 for estimates for subgroups of UBMS participants. 



While we cannot measure the extent of selection bias, some informed speculation is helpful 
in interpreting the impact estimates. As we indicated earlier, students in both groups — the 
UBMS participant sample and the matched comparison group — exhibited some motivation to 
improve academically by applying to participate in Upward Bound. Therefore, motivational 
differences between the two groups are likely to be small and unlikely to bias the impact 
estimates. However, the impact estimates would overstate the true impacts if: 



• Members of UBMS participant sample were higher achievers than members of 

matched comparison group, by the end of high school and after students in the 

UBMS sample had participated in UBMS, the average GPA was slightly but 
significantly higher for UBMS participants (3.14) than for matched comparison 
students (3.06). Even if the entire difference were attributable to selection bias 
instead of the effects of the program, the average student in each group was a B 
student. Therefore, if the UBMS participant sample contains higher achieving 
students than the matched comparison group, the difference seems to be small. 
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• Members of the UBMS participant sample were better at math and science than 
members of the matched comparison group. If students in the UBMS participant 
sample were better at math and science than students in the matched comparison 
group, it is not reflected in their math and science course-taking or grades during 
ninth grade: the two groups had similar grades in math and science, were equally 
likely to take algebra or geometry and were equally likely to take biology, chemistry 
or physics. While we did not explicitly examine whether UBMS participants were 
more likely to enroll in the advanced sections of courses before entering UBMS, 
other findings suggest that UBMS participants were no more likely to have taken 
Advanced Placement classes in math or science by the time they finished high 
school. Finally, focus groups conducted in 1996 suggest that UBMS participants do 
not view themselves as particular strong in math and science (Moore 1997, p. 28). 
Therefore, the information we have collected provides no reason to believe that when 
they entered UBMS, the participant sample was better at math and science than the 
matched comparison group. 

• Members of the UBMS participant group were more interested in math and 
science than members of the matched comparison group. None of the infonnation 
that we extracted from student transcripts would suggest that the UBMS participant 
sample had greater interest in math and science. It is possible UBMS participants 
had greater interest in careers in math and science that simply was not reflected in 
their high school course-taking or grades early in high school, in 1996 focus groups, 
many UBMS participants expressed interest in pursuing careers in scientific fields, 
such as engineering, medicine, and nursing (Moore 1997, p. 28). However, many of 
these students indicated that their career interests had developed just that summer, 
and the expression of those interests could have been influenced by the fact that the 
focus groups were conducted on site at projects that emphasized math and science. 

We suspect that the impacts most vulnerable to selection bias are those that are most closely 
related to a person’s interest in pursuing careers in a math or science field. To gauge students’ 
interest in math and science early in high school would probably require conducting assessments 
or survey interviews at that time. However, we first interviewed members of the UBMS 
participant sample after they had completed high school. The possibility that UBMS participants 
might have had greater interest in pursuing a career in science than matched comparison students 
raises the question of whether our estimates overstate the effects of UBMS on the outcomes that 
are most closely related to one’s career interests, such as majoring in math or science in college. 

Therefore, while the findings in this report are promising, a note of caution is appropriate. 
We speculate that the selection bias is likely to be largest for outcome variables most closely tied 
one’s interest in pursuing math and science careers, but it is not possible to measure the selection 
bias. While we took several steps to reduce selection bias, the estimated effects of UBMS may 
overstate the true effects of the program. 
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APPENDIX A 



DATA COLLECTION 




This appendix describes and assesses the procedures for collecting the data that we used to 
construct student outcome measures for the impact analysis presented in Chapter III. These data 
come from two different sources: 



1 . The fourth follow-up survey of students 

2. Secondary and postsecondary transcripts 



This appendix focuses on procedures for obtaining 
follow-up survey and for collecting academic transcripts. 



completed interviews in the fourth 



A. Fourth Follow-Up Survey of Students 

The fourth follow-up survey was conducted between April 2001 and December 2002. It was 
designed to collect information on secondary and postsecondary educational outcomes 
approximately five to seven years after scheduled completion of high school. 



1. Data Collection Modes 

One week before we began interviewing, we sent a letter to all study participants. The letter 
indicated that we would call them to complete an interview for an important study, and it 
encouraged them to participate. In addition, the letter indicated that we would pay them $10 for 
completing the interview. 

Most interviews were administered using computer-assisted telephone interviewing (CATI). 
CATI interviews took about 30 minutes to complete. When a CATI interview was not possible, 
we attempted to obtain a completed questionnaire through the mail. Study participants were also 
offered the option of completing the survey on the Web. In June 2001, questionnaires were 
mailed to study participants that could not be reached by telephone. Three additional follow up 
mailings were conducted after the first mailing, with the last set of questionnaires being sent out 
in January 2002. 



2. Locating 

Throughout the data collection period, locating staff used services such as LexisNexis and 
Internet databases to obtain updated addresses and phone numbers for study participants that 
were difficult to reach. 



3. Incentives 

Financial incentives for survey completion were used to obtain a high response rate. Study 
participants were offered a $10 incentive for participating in the survey. Incentive checks were 
mailed after the sample member completed the interview. 
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4. Response Rates 



The eligible sample consisted of 1,759 UBMS participants and 2,830 sample members from 
the evaluation of regular Upward Bound. (See Chapter III for more details on the samples.) We 
obtained completed interviews for 1,425 UBMS participants and 2,146 regular Upward Bound 
sample members for response rates of 81 percent and 76 percent, respectively (see Table A.l). 



B. Transcript Data Collection 

Secondary and postsecondary transcripts were collected between July 2002 and March 2003. 
Academic transcripts provided the primary source of information on postsecondary achievement. 
Transcript requests were made from institutions that were reported by sample members in the 
fourth follow-up survey of students and in earlier surveys. 



1. Preparation for Requesting Transcripts 

Information about students’ secondary and postsecondary enrollment was primarily obtained 
from follow-up interviews. Students reported the secondary and postsecondary institutions that 
they had attended. Secondary transcripts were only requested from UBMS sample members 
selected for the impact analysis; postsecondary transcripts were requested from all sample 
members — both UBMS and regular Upward Bound — who reported or confirmed having 

48 

attended a particular postsecondary institution. 

To obtain mailing addresses for the schools that were attended by sample members, we 
matched schools that were reported by survey respondents to directories of secondary and 
postsecondary schools maintained by the U.S. Department of Education. Secondary schools 
were matched to the Common Core of Data (CCD); postsecondary schools were matched to the 
integrated Postsecondary Education Data System (IPEDS). 49 



48 We did not collect secondary transcripts for regular Upward Bound sample members because we had already 
collected these transcripts for a large percentage of the sample in previous waves of data collection. 

49 Students were asked to provide the name and state of each secondary and postsecondary school they 
attended, but sometimes misspellings or incomplete information resulted in some invalid requests for student 
transcripts as schools were matched with an incorrect address and transcripts were requested from the wrong school. 
When a school indicated that they could not fill a request because they had no record of the student whose transcript 
we requested, it was sometimes due to such mismatches. In these cases, we attempted to learn the correct name and 
address of the school where the student was enrolled and make a new transcript request. 
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Table A.l 

Fourth Follow-up Survey of Students 





UBMS Sample 


Regular UB Sample 


Full Sample 


Completed Interview 


1,425 


2,146 


3,571 


Eligible Nonrespondent 


334 


684 


1,018 


Ineligible - Deceased 


7 


14 


21 


Total 


1,766 


2,844 


4,610 



2. Procedures for Requesting Transcripts 

Each school was sent a transcript request packet that included: 

• A letter, printed on Department of Education letterhead, which explained the 
purpose of the study and the reason we were requesting transcripts. 

• A statement of Authorization and Confidentiality, which cited the Family 
Educational Rights and Privacy Act and included questions and answers regarding 
consent and confidentiality. 

• A transcript checklist of all the materials that we requested from the school, 
including student transcripts, a course catalog, grade descriptions and a transcript 
reimbursement form, which would indicate the reimbursement that the school 
required for providing the requested transcripts. 

• A postage-paid business reply envelope for sending the transcripts. 

• A disclosure notice to be placed in each student’s file, indicating that a copy of his 
or her transcript was released to Mathematica Policy Research as an agent to the 
U.S. Department of Education. 

3. Follow-Up Procedures 

For schools that did not respond to our initial request for transcripts, we mailed another 
request for student transcripts. These mailings were done periodically as we tracked the schools 
that had not yet sent the requested transcripts and corrected requests that contained errors. 

As the targeted end date for collecting transcripts approached, interviewers started calling 
schools directly to inquire about the status of our requests. Many schools responded to these 
calls by faxing us the requested transcripts. When the school indicated that they could not 
provide one or more of the requested transcripts, the interviewer completed a problem sheet 
indicating the reason. The reason generally fell into one of the following categories: 
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• The student was never enrolled at the school according to the school’s records. 
When this occurred, our first response was to call the school and provide more 
information on the student (e.g., provide or verify date of birth and dates of 
attendance) to see if a transcript could be located with additional infonnation. In 
many cases, the school was able to locate and provide transcripts once additional 
information was provided. In other cases, the school provided some information 
that helped us determine where we might obtain the needed transcripts. 50 If the 
school had no record of the student having ever attended and we were unable to 
obtain additional information, we marked the case as an invalid request. 

• Transcripts were held by the school district. Some schools only held the 
transcripts of currently enrolled students and all other transcripts were sent to the 
school district, in this situation, the school would sometimes forward the request 
packet to the district. Other times, the school returned the materials to us, and we 
sent them to the school district. 

• The student transferred to another school. When the student had transferred to 
another school, a transcript was requested from the school to which the student had 
transferred. In some cases, the registrar or school secretary forwarded the request 
materials to the transfer school, in other cases, the request materials were sent back 
to us and we sent a new request to the transfer school. 

• The school would not release any transcript without student’s written consent. A 
few schools returned the transcript request materials with no transcripts, indicating 
that they required written consent from each student whose transcript we were 
requesting. A problem sheet was completed for these cases, and they were 
forwarded to the survey manager for follow-up. As a first step, the survey manager 
called the school to explain that, as an agent of the Department of Education, 
Mathematica Policy Research was authorized to collect student transcripts for the 
purposes of this study and that, according to the laws of FERPA, schools are 
pennitted to release student transcripts to the Department of Education without the 
written consent of students participating in the study. It was also explained that 
students had given verbal consent over the telephone or written consent when they 
completed the mail survey, and that we did not request transcripts for any students 
who refused consent. Some schools agreed to send the requested transcripts upon 
hearing this explanation. Others reiterated that signed consent was required by 
school policy, in this case, we sent written consent forms to the students for them 
to sign and return to Mathematica so that we could obtain their student transcript 
for the impact study. A postage-paid return envelope was included with the 
consent form. A small number of students did sign and return the consent form, 
but most of the letters came back unopened because we no longer had a valid 
address for the student. 



50 For example, some school principals and registrars indicated that their school was often confused with 
another school having the same or a similar name and suggested that we direct our request to the other school. In 
this case, we would call the alternate school to find out if the student was ever enrolled there. If so, we made a 
correction to the database and sent a request to the newly identified school. 
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• The school would not release transcripts without advance payment. In these 
cases, we sent a check to cover the cost of each transcript, along with a list of the 
students whose transcripts we were requesting. 

• The school would not release a transcript until the student paid an outstanding 
debt. In some cases we were eventually able to obtain these transcripts as students 
paid whatever bills they owed the school. When the debt remained unpaid, 
however, there was no way we could get the transcript. These cases were marked 
as unfdled requests. 

4. Response Rates 

From the samples used in the impact analysis described in Chapter III, 1,365 students 
reported having attended at least one postsecondary institution that could be matched to IPEDS. 
for each of these 1,365 students, we requested a transcript from each of the postsecondary 
institutions that he or she reported attending. In total, we requested 2,029 transcripts. We 
received 1,821 of the 2,029 transcripts requested (90 percent), and we obtained a complete 
transcript record — that is, transcripts for all postsecondary institutions attended — for 1,109 
students (see Table A. 2). 



Table A.2 

Postsecondary Transcript Data Collection for the Upward Bound Math-Science Impact Analysis 



Postsecondary Students 


UBMS Sample 


Regular UB Sample 


Full Sample 


Complete Transcript Record 


472 


637 


1,109 


Incomplete Transcript Record 


101 


155 


256 


Total Postsecondary Students 


573 


792 


1,365 
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APPENDIX B 



PROGRAM IMPACTS 




Table B.l 

Impact of Upward Bound Math Science on High School Outcomes 





UBMS Participants 


Comparison Group 


Impact 


Overall 


Total Credits 


25.7 


24.5 


1 2 *** 


Grade Point Average (GPA) 


3.1 


3.1 


Q 1 *** 


Math Courses Taken 


Algebra (%) 


97 


97 


0 ! 


Geometry (%) 


86 


86 


0 


Algebra II (%) 


73 


74 


-1 


Trigonometry (%) 


35 


30 


5 


Analysis / Precalculus (%) 


36 


31 


5 


Calculus (%) 


26 


24 


2 


Advanced Placement Calculus (%) 


6 


10 


-3 * 


GPA in Algebra or above 


2.8 


2.7 


0 i *** 


Science Courses Taken 


Biology (%) 


97 


98 


-1 


Chemistry (%) 


88 


78 


*** 


Physics (%) 


58 


43 


15 *** 


Advanced Placement Biology, Chemistry, or 


Physics (%) 


4 


9 


-5 ! 


GPA in Biology, Chemistry, and Physics 


2.9 


2.7 


0 2 *** 


High School Status (%) 


Graduated 


99 


96 


2 *** 


Dropped out 


1 


2 


-2 ! 


General Educational Development (GED) 


0 


1 


-1 ! 



Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table B.2 

Impact of Upward Bound Math Science on High School Outcomes by Prior Participation in Regular Upward Bound 





Participated 


in Regular Upward Bound 


Did Not Participate in Regular Upward Bound 




UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


Overall 


Total Credits 


24.6 


24.5 


0.1 


26.0 


24.5 




Grade Point Average (GPA) 


3.1 


3.0 


0.0 


3.2 


3.1 


o i *** 


Math Courses Taken 


Algebra (%) 


97 


98 


-1 ! 


97 


97 


1 ! 


Geometry (%) 


88 


88 


0 


86 


85 


1 


Algebra II (%) 


66 


78 


_12 ** 


74 


73 


1 # 


Trigonometry (%) 


38 


27 


II ** 


34 


30 


4 


Analysis / Precalculus (%) 


34 


29 


5 


37 


32 


5 


Calculus (%) 


22 


21 


2 


27 


25 


2 


Advanced Placement Calculus (%) 


6 


7 


-2 


7 


10 


-3 * 


GPA in Algebra or above 


2.7 


2.6 


0.1 


2.8 


2.7 


0 i *** 


Science Courses Taken 


Biology (%) 


98 


97 


1 ! 


97 


98 


-1 


Chemistry (%) 


91 


78 


12 *** 


88 


79 


9 *** 


Physics (%) 


53 


41 


12 ** 


59 


43 




Advanced Placement Biology, Chemistry, or 


Physics (%) 


2 


5 


-4 ! 


5 


10 


-5 ** 


GPA in Biology, Chemistry, and Physics 


2.8 


2.7 


0.1 ** 


2.9 


2.7 


Q 2 *** 


High School Status (%) 


Graduated 


100 


99 


1 ! 


99 


96 


2 ** 


Dropped out 


0 


1 


-1 ! 


1 


3 


-2 ! 


General Educational Development (GED) 


0 


0 


0 ! 


1 


1 


-1 ! 



Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on students who participated in regular Upward Bound at the 0.10 level. 
! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table B.3 

Impact of Upward Bound Math Science on High School Outcomes hy Sex 



Male Female 

UBMS Comparison UBMS Comparison 

Participants Group Impact Participants Group Impact 



Overall 

Total Credits 

Grade Point Average (GPA) 

Math Courses Taken 

Algebra (%) 

Geometry (%) 

Algebra II (%) 

Trigonometry (%) 

Analysis / Precalculus (%) 

Calculus (%) 

Advanced Placement Calculus (%) 

GPA in Algebra or above 

Science Courses Taken 

Biology (%) 

Chemistry (%) 

Physics (%) 

Advanced Placement Biology, Chemistry, or 
Physics (%) 

GPA in Biology, Chemistry, and Physics 

High School Status (%) 

Graduated 
Dropped out 

General Educational Development (GED) 



25.3 


24.6 


0.7 


3.1 


3.0 


0.1 * 


97 


98 


0 ! 


84 


87 


-3 


77 


73 


4 


33 


34 


-1 


42 


35 


7 


25 


22 


3 


7 


9 


-2 


2.7 


2.6 


0.1 


98 


99 


-1 ! 


87 


77 


1 1 *** 


60 


48 


12 ** 


5 


7 


-2 ! 


2.9 


2.6 


0 3 *** 


98 


98 


0 ! 


1 


1 


0 ! 


1 


1 


0 ! 



26.1 


24.5 


1 ^ *** 


3.2 


3.1 


o 1 *** 


97 


97 


0 ! 


88 


84 


3 


70 


74 


-5 


36 


28 


8 * 


32 


28 


4 


27 


25 


2 


6 


10 


-4 


2.8 


2.7 


o 1 *** 


97 


98 


-1 


89 


80 


9 *** 


56 


39 


jg *** 


4 


10 


-6 ! 


2.9 


2.8 


0 9 *** 


99 


96 


4 ! 


0 


2 


-2 ! 


0 


2 


-1 ! 



Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on males at the 0.10 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table B.4 

Impact of Upward Bound Math Science on High School Outcomes by Race and Ethnicity 





African American 






White 






Hispanic 






UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


Overall 


Total Credits 


25.5 


24.1 


\ 4 ** 


26.0 


25.2 


0.8 


26.4 


24.6 


1.9 ** 


Grade Point Average (GPA) 


2.9 


2.9 


0.1 * 


3.3 


3.3 


0.0 


3.2 


3.0 


0.2 *** # 


Math Courses Taken 


Algebra (%) 


96 


97 


-1 ! 


100 


99 


1 ! 


96 


95 


0 ! 


Geometry (%) 


88 


89 


-1 


91 


87 


5 ** 


81 


79 


2 


Algebra II (%) 


75 


75 


0 


79 


81 


-2 ! 


67 


74 


-7 


Trigonometry (%) 


35 


32 


2 


38 


31 


7 


35 


24 


11 * 


Analysis / Precalculus (%) 


33 


27 


5 


30 


22 


8 


42 


30 


12 


Calculus (%) 


17 


16 


1 


27 


27 


1 


29 


21 


9 ! 


Advanced Placement Calculus (%) 


4 


4 


0 ! 


4 


12 


-8 ! 


9 


8 


1 ! 


GPA in Algebra or above 


2.5 


2.4 


0.1 


3.2 


3.1 


0.1 


2.7 


2.4 


0 3 *** # 


Science Courses Taken 


Biology (%) 


97 


99 


-2 


98 


98 


0 ! 


98 


96 


2 ! 


Chemistry (%) 


90 


83 


7 ** 


87 


77 


10 ! 


87 


70 


17 *** ^ 


Physics (%) 


53 


46 


7 


59 


36 


22 *** # 


59 


32 


27 *** # 


Advanced Placement Biology, Chemistry, or 


Physics (%) 


4 


7 


-3 ! 


2 


6 


-4 ! 


6 


12 


-6 ! 


GPA in Biology, Chemistry, and Physics 


2.7 


2.5 


q 2 *** 


3.3 


3.0 


Q ^ *** 


2.9 


2.7 


0.2 ** 


High School Status (%) 


Graduated 


100 


97 


3 ! 


99 


97 


2 ! 


98 


96 


3 ! 


Dropped out 


0 


2 


-2 ! 


0 


2 


-2 ! 


2 


4 


-3 ! 


General Educational Development (GED) 


0 


0 


0 ! 


1 


1 


0 ! 


0 


2 


-2 ! 



Source: MPR analysis file "UBMSImpact6.sas." 

*/♦*/*** j m p aC { estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on African Americans at the 0.10 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 




Table B.5 

Impact of Upward Bound Math Science on Postsecondary Outcomes 



UBMS Participants Comparison Group Impact 



Postsecondary Enrollment (%) 

Any postsecondary institution 
Highest level of schooling attended 
Four-year college or university 
Two-year college 
Vocational institution 

College Selectivity (%) 

Most selective four-year college or university 
More selective 
Less selective 

Postsecondary Credits Earned (mean) 

Two- and four-year colleges and universities 
Four-year colleges and universities 
Two-year colleges and universities 

Postsecondary Completion (%) 

Any degree, certificate, or license 
Highest degree, certificate, or license earned 
Bachelor's degree or higher 
Associate's degree 
Certificate or license 



95 


90 


5 *** 


82 


71 




11 


16 


-5 ** 


2 


4 


-2 



33 


23 


1 1 *** 


48 


48 


0 


98.1 


85.0 


1 2 1 *** 


86.6 


73.1 


13.5 ** 


11.5 


11.9 


-0.5 


47 


46 


1 


35 


33 


2 


7 


7 


-1 


5 


5 


-1 



Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table B.6 

Impact of Upward Bound Math Science on Postsecondary Outcomes by Prior Participation in Regular Upward Bound 





Participated 


in Regular Upward Bound 


Did Not Participate in Regular Upward Bound 




UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


Postsecondary Enrollment (%) 


Any postsecondary institution 


99 


91 


g *** 


95 


90 


5 *** 


Highest level of schooling attended 


Four-year college or university 


81 


71 


10 * 


82 


70 


12 *** 


Two-year college 


18 


16 


1 


10 


16 


-6 ** 


Vocational institution 


0 


4 


-4 ! 


3 


4 


-1 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


35 


24 




33 


22 


l 1 *** 


Less selective 


46 


47 


-1 


48 


48 


0 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


103.4 


84.2 


19.2 ** 


96.9 


85.2 


1 1 7 ** 


Four-year colleges and universities 


90.1 


69.2 


20.9 ** 


85.9 


74.3 


11.5 * 


Two-year colleges and universities 


13.3 


15.0 


-1.7 


11.1 


10.9 


0.2 


Postsecondary Completion (%) 


Any degree, certificate, or license 


42 


43 


-2 


48 


46 


2 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


31 


29 


2 


36 


34 


2 


Associate's degree 


7 


5 


2 


7 


8 


-1 


Certificate or license 


4 


8 


-4 


5 


5 


0 



Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** j m p ac t estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on students who participated in regular Upward Bound at the 0.10 level. 
! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table B.7 

Impact of Upward Bound Math Science on Postsecondary Outcomes by Sex 







Male 






Female 








UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 




Postsecondary Enrollment (%) 


Any postsecondary institution 


96 


89 




95 


90 


5 *** 




Highest level of schooling attended 


Four-year college or university 


82 


76 


6 


82 


68 


14 *** 


# 


Two-year college 


12 


12 


0 


10 


18 


_g *** 


# 


Vocational institution 


2 


2 


0 


2 


4 


-2 




College Selectivity (%) 

Most selective four-year college or university 


More selective 


34 


23 


1 1 ** 


32 


21 


1 1 *** 




Less selective 


48 


52 


-5 


48 


46 


2 




Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


96.6 


85.7 


10.9 


99.1 


85.2 


1 3 g *** 




Four-year colleges and universities 


83.2 


73.5 


9.7 


88.9 


72.7 


16.2 *** 




Two-year colleges and universities 


13.4 


12.2 


1.2 


10.1 


12.5 


-2.3 




Postsecondary Completion (%) 


Any degree, certificate, or license 


40 


43 


-3 


51 


46 


6 




Highest degree, certificate, or license earned 


Bachelor's degree or higher 


27 


33 


-6 


40 


32 


g ** 


# 


Associate's degree 


8 


4 


4 * 


6 


10 


_4 * 


# 


Certificate or license 


5 


5 


0 


5 


5 


0 





Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0. 10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on males at the 0. 10 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table B.8 

Impact of Upward Bound Math Science on Postsecondary Outcomes by Race and Ethnicity 





African American 






White 






Hispanic 






UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


Postsecondary Enrollment (%) 


Any postsecondary institution 


97 


93 


5 ! 


95 


87 


8 ! 


95 


92 


2 ! 


Highest level of schooling attended 


Four-year college or university 


87 


74 


[3 *** 


79 


67 


12 ! 


80 


72 


8 ! 


Two-year college 


9 


15 


-6 


12 


19 


-7 ! 


14 


17 


-3 ! 


Vocational institution 


1 


3 


-2 ! 


5 


1 


4 ! 


0 


6 


-6 ! 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


29 


19 


10 ** 


24 


11 


13 *** 


38 


36 


3 


Less selective 


57 


54 


3 


54 


55 


0 


42 


35 


7 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


105.9 


87.0 


18.9 ** 


92.9 


85.8 


7.2 


97.6 


65.7 


31 g *** 


Four-year colleges and universities 


99.4 


79.2 


20.2 ** 


81.6 


73.3 


8.4 


81.2 


52.9 


28.3 *** 


Two-year colleges and universities 


6.5 


7.8 


-1.3 


11.3 


12.5 


-1.2 


16.4 


12.8 


3.6 


Postsecondary Completion (%) 


Any degree, certificate, or license 


49 


46 


3 


46 


35 


10 ** 


49 


49 


-1 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


38 


36 


1 


36 


19 


17 *** $ 


33 


30 


3 


Associate's degree 


5 


4 


2 ! 


8 


10 


-3 


8 


13 


-5 


Certificate or license 


6 


6 


-1 ! 


2 


6 


-4 ! 


8 


5 


3 



Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on African Americans at the 0.10 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table B.9 

Impact of Upward Bound Math Science on Postsecondary Field of Study 



Field of Study 


UBMS Participants 


Comparison Group 


Impact 


Field of Study at Most Recent PS Institution (%) 

All postsecondary institutions 


Math and science fields 


33 


23 


io *** 


Social science fields 


11 


7 


4 * 


Other fields 


42 


51 


_g *** 


Four-year colleges and universities 


Math and science fields 


28 


18 


jo *** 


Social science fields 


10 


7 


4 * 


Other fields 


36 


39 


-3 


Earned Degree or Certificate in Field (%) 

All postsecondary institutions 


Math and science fields 


15 


12 


3 


Social science fields 


7 


4 


3 


Other fields 


21 


28 




Four-year colleges and universities 


Math and science fields 


12 


6 




Social science fields 


6 


4 


3 


Other fields 


14 


20 





Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table B. 10 

Impact of Upward Bound Math Science on Postsecondary Field of Study by Prior Participation in Regular Upward Bound 





Participated 


in Regular Upward Bound 


Did Not Participate in Regular Upward Bound 


Field of Study 


UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


Field of Study at Most Recent PS Institution (%) 

All postsecondary institutions 


Math and science fields 


35 


26 


10 * 


32 


22 


10 ** 


Social science fields 


13 


7 


6 * 


11 


8 


3 


Other fields 


38 


47 


-9 


43 


52 


_g ** 


Four-year colleges and universities 


Math and science fields 


29 


23 


6 


28 


17 


12 *** 


Social science fields 


12 


6 


6 


10 


7 


3 


Other fields 


31 


35 


-3 


37 


40 


-4 


Earned Degree or Certificate in Field (%) 

All postsecondary institutions 


Math and science fields 


13 


12 


1 


15 


11 


4 


Social science fields 


6 


3 


3 


7 


4 


3 


Other fields 


20 


24 


-4 


22 


29 


JJ ** 


Four-year colleges and universities 


Math and science fields 


8 


10 


-2 ! 


13 


5 


g *** 


Social science fields 


4 


3 


1 


7 


4 


3 


Other fields 


15 


13 


2 


14 


22 


-8 ** # 



Source: UBMSlmpact6.sas 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on students who participated in regular Upward Bound at the 0.10 level. 
! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table B.ll 

Impact of Upward Bound Math Science on Postsecondary Field of Study by Sex 







Male 






Female 






Field of Study 


UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 




Field of Study at Most Recent PS Institution (%) 

All postsecondary institutions 


Math and science fields 


44 


28 


*** 


24 


19 


5 




Social science fields 


9 


5 


4 


12 


9 


3 




Other fields 


31 


46 


*** 


50 


53 


-3 


# 


Four-year colleges and universities 


Math and science fields 


38 


25 


13 ** 


22 


13 


g ** 




Social science fields 


8 


4 


5 * 


12 


9 


3 




Other fields 


27 


38 


-12 ** 


42 


39 


4 


# 


Earned Degree or Certificate in Field (%) 

All postsecondary institutions 


Math and science fields 


16 


10 


6 * 


14 


11 


3 




Social science fields 


5 


3 


2 ! 


8 


5 


3 * 




Other fields 


15 


25 


-10 ** 


26 


28 


-2 




Four-year colleges and universities 


Math and science fields 


11 


6 


5 * 


13 


5 


g *** 




Social science fields 


4 


3 


1 ! 


8 


4 






Other fields 


9 


19 


*** 


18 


20 


-2 


# 



Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on males at the 0. 10 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table B. 12 

Impact of Upward Bound Math Science on Postsecondary Field of Study by Race and Ethnicity 





African American 






White 






Flispanic 




Field of Study 


UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


Field of Study at Most Recent PS Institution (%) 

All postsecondary institutions 


Math and science fields 


32 


25 


7 


34 


26 


8 


28 


14 


15 ** 


Social science fields 


13 


6 


6 ** 


9 


4 


5 ! 


12 


8 


4 ! 


Other fields 


42 


59 


*** 


42 


50 


-8 


48 


55 


-7 


Four-year colleges and universities 


Math and science fields 


30 


20 


10 * 


27 


20 


7 


26 


8 


18 ** 


Social science fields 


13 


6 


y ** 


8 


4 


4 ! 


8 


7 


1 ! 


Other fields 


38 


44 


-7 


35 


38 


-4 


40 


45 


-5 


Earned Degree or Certificate in Field (%) 

All postsecondary institutions 


Math and science fields 


14 


10 


4 ! 


16 


9 


6 


15 


9 


7 


Social science fields 


8 


4 


4 * 


5 


2 


4 ! 


5 


5 


0 ! 


Other fields 


21 


32 


1 ** 


21 


20 


1 # 


26 


30 


-4 


Four-year colleges and universities 


Math and science fields 


13 


6 


7 ! 


13 


5 


8 ! 


11 


2 


9 ! 


Social science fields 


7 


3 


4 * 


5 


2 


4 ! 


5 


5 


0 ! 


Other fields 


15 


23 


-8 * 


13 


9 


4 # 


15 


19 


-4 



Source: MPR analysis file "UBMSImpact6.sas." 

♦/**/*** j m p ac t estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on African Americans at the 0.10 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 




APPENDIX C 



SENSITIVITY OF IMPACTS TO AN ALTERNATIVE MEASURE OF 
POSTSECONDARY ATTENDANCE 




A. Verification of Students’ Postsecondary Attendance 



In Chapter III, we presented estimates of the effects of UBMS on postsecondary attendance 
based on self-reported attendance — specifically, the schools that sample members reported 
attending after high school. However, in some instances, sample members may not have actually 
attended the schools they reported. For example, some may have reported schools that they 
planned to attend but never attended, and others may have reported schools at which they 
participated in a noncredit program but were not enrolled as students. In this appendix, we 
conduct an analysis to determine if the impact estimates presented in Chapter III are sensitive to 
an alternative measure of postsecondary attendance, one that requires verification by the 
postsecondary institution that students reported attending. 

Myers et al. (2004) present the results of this analysis in the text of the report along with the 
analysis based on self-reported data, which effectively provided two impact estimates for each 
outcome. When one of the two impact estimates was statistically insignificant, the results were 
characterized as inconclusive because at that time, we were unable to persuade ourselves that one 
method was more reliable than the other. However, we have since conducted an investigation of 
the verification process, and the results of this analysis indicate that on average, the self-reported 
measures of postsecondary attendance are more accurate than the measures requiring verification 
by the institution. While this may seem counterintuitive, in constructing postsecondary 
attendance measures that required verification, we assumed that students did not attend the 
schools they reported unless attendance was verified by the school, typically through the 
provision of a transcript. However, the transcript collection process was not originally designed 
to verify attendance. 51 And in most instances when a school was unable or unwilling to provide 
a transcript, a careful examination of the data suggests that the student probably did attend the 
school — or at the very least, the school gave us no reason to doubt the student had attended when 
explaining why they would not provide a transcript. 

The verification process worked as follows. In the third and fourth follow-up surveys, 
students reported all postsecondary institutions that they had attended since high school. We 
then attempted to match all reported schools to the 1997-98 Integrated Postsecondary Education 
Data System (IPEDS) maintained by the National Center for Education Statistics (NCES) to 
determine whether they met NCES’s definition of a postsecondary institution and to obtain 
school contact information. Transcript requests were then sent to all schools a student reported 
attending that we were able to match to IPEDS and schools that did not initially respond were 
followed up. If a school provided a transcript that we requested for a student in the sample, then 
the student clearly attended that school. Furthennore, some of the reasons given by school staff 
for not providing transcripts can be treated as verification of attendance, for example, a college 
indicating that it could not provide a transcript for a student because he or she owed money to the 
school. In many instances, however, the reason given for not providing transcripts does not 
clearly indicate whether the student attended the school. For example, some schools required 
written consent from the students themselves even though the law does not require it, and we 
typically obtained only oral consent (see Appendix B); in these cases, lack of verification casts 
no doubt on the student’s self-reported attendance. 



51 For the final report of this evaluation, which will use postsecondary transcript data collected in 2004-05, the 
transcript data collection process has been modified to specifically verify attendance if possible. 
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The information obtained while collecting transcripts is therefore useful in verifying 
attendance in some but not all cases. In the vast majority of cases considered (approximately 80 
percent) we were able to verify the student’s attendance. However, for approximately 220 
students, the fact that we were unable to verify attendance for at least one of their reported 
postsecondary institutions changed at least one outcome of interest. In investigating these 
cases, we found little evidence that contradicted a student’s report of their postsecondary schools 
attended. One concern was that some Upward Bound participants would report the school at 
which they attended an Upward Bound summer bridge program but did not actually enroll; we 
found no evidence that this happened.' 23 Similarly, there was little evidence that many students 
reported attending institutions that are not classified by IPEDS as postsecondary institutions, and, 
in fact, non-IPEDS institutions such as Job Corps programs were excluded through the IPEDS 
match. The lack of verification more often appeared to be due to schools that would not release 
transcripts without written consent, ambiguities in the exact campus attended by the student 
within a large state system, or insufficient information about the school reported by the student to 
know which school to contact for transcript information. More often than not, lack of 
verification seemed to mainly reflect limitations of the verification process rather than 
inaccuracies in the infonnation provided by students. 



B. Differences in Impacts 

This section presents estimates of the impacts of Upward Bound that use information only 
on schools that we were able to verify the student attended. These estimates make the strong 
assumption that lack of verification of attendance implies that the student did not attend the 
school in question or any school like it. 

Because most attendance reported by students was verified through transcript receipt, most 
of the impacts presented here are very similar to those estimated using information on all schools 
that students reported attending. There are a few differences between the estimates presented 
here and the estimates presented in Chapter III, but there is probably only one worth noting: the 
finding that UBMS has a positive effect on the likelihood of attending some type of 
postsecondary institution is sensitive to the verification process. If we require verification of 
postsecondary attendance, the estimated effect becomes statistically insignificant (compare Table 

C. l to Table B.5). For the full set of estimates from the sensitivity analysis, see Tables C.l - C.4. 



52 There may be cases in which we were not able to verify a student’s attendance at all schools but for which 
that did not affect the outcome variables of interest. For example, if we verified a student’s attendance at a four-year 
college or university but could not verify his or her attendance at a vocational school they reported attending, he or 
she will still be classified as having their highest level of schooling as a four-year college or university. 

53 Students participate in Upward Bound summer bridge programs the summer before enrolling in college. 
Because the survey question simply asks students to list all postsecondary institutions attended since high school 
graduation, there was concern that some students may list the school at which they participated in a summer bridge 
program. 
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Table C.l 



Impact of Upward Bound Math Science on Postsecondary Outcomes, 
Excludes Unverified Enrollment and Completion 





UBMS Participants 


Comparison Group 


Impact 


Postsecondary Enrollment (%) 


Any postsecondary institution 


89 


87 


2 


Highest level of schooling attended 


Four-year college or university 


77 


68 


g *** 


Two-year college 


11 


16 


-5 ** 


Vocational institution 


1 


3 


-2 * 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


30 


21 




Less selective 


46 


47 


-1 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


84.8 


75.2 


9.6 ** 


Four-year colleges and universities 


75.3 


64.8 


10.5 ** 


Two-year colleges and universities 


9.6 


10.4 


-0.8 


Postsecondary Completion (%) 


Any degree, certificate, or license 


43 


43 


0 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


33 


32 


1 


Associate's degree 


6 


7 


-1 


Certificate or license 


4 


4 


1 



Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table C.2 



Impact of Upward Bound Math Science on Postsecondary Outcomes by Prior Participation in Regular Upward Bound, 

Excludes Unverified Enrollment and Completion 





Participated 


in Regular Upward Bound 


Did Not Participate in Regular Upward Bound 




UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


Postsecondary Enrollment (%) 


Any postsecondary institution 


91 


86 


4 


89 


87 


2 


Highest level of schooling attended 


Four-year college or university 


75 


67 


8 


77 


68 


9 *** 


Two-year college 


16 


17 


0 


10 


16 


_5 ** 


Vocational institution 


0 


2 


-2 ! 


1 


3 


-2 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


31 


23 


8 


30 


20 


|Q *** 


Less selective 


44 


43 


0 


46 


47 


-1 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


83.5 


65.7 


17 g *** 


85.1 


77.1 


8.1 


Four-year colleges and universities 


72.8 


53.9 


18.9 ** 


75.8 


67.0 


8.8 


Two-year colleges and universities 


10.7 


11.7 


-1.1 


9.3 


10.1 


-0.7 


Postsecondary Completion (%) 


Any degree, certificate, or license 


36 


41 


-6 


45 


43 


2 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


25 


28 


-2 


35 


33 


2 


Associate's degree 


6 


5 


1 


6 


7 


-1 


Certificate or license 


4 


7 


-2 


4 


3 


1 



Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on students who participated in regular Upward Bound at the 0.10 level. 
! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table C.3 



Impact of Upward Bound Math Science on Postsecondary Outcomes by Sex, 
Excludes Unverified Enrollment and Completion 







Male 






Female 








UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 




Postsecondary Enrollment (%) 


Any postsecondary institution 


88 


88 


1 


89 


85 


4 * 




Highest level of schooling attended 


Four-year college or university 


74 


74 


0 


78 


64 


15 *** 


# 


Two-year college 


13 


13 


0 


10 


18 




# 


Vocational institution 


1 


1 


0 ! 


1 


4 


-3 * 




College Selectivity (%) 

Most selective four-year college or university 


More selective 


30 


21 


9 ** 


30 


19 


|Q *** 




Less selective 


43 


53 


_9 * 


47 


44 


3 


# 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


80.2 


76.3 


3.9 


88.1 


73.9 


14 2 *** 




Four-year colleges and universities 


69.7 


67.6 


2.0 


79.2 


62.0 


12 ^ *** 


# 


Two-year colleges and universities 


10.6 


8.7 


1.9 


8.9 


11.9 


-3.1 




Postsecondary Completion (%) 


Any degree, certificate, or license 


37 


42 


-5 


48 


43 


6 


# 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


26 


33 


-7 


38 


31 


2 * 


# 


Associate's degree 


7 


3 


4 * 


6 


10 


_4 * 


# 


Certificate or license 


4 


4 


0 


4 


3 


i 





Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on males at the 0.10 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 
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Table C.4 



Impact of Upward Bound Math Science on Postsecondary Outcomes by Race and Ethnicity, 
Excludes Unverified Enrollment and Completion 





African American 






White 






Hispanic 






UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


UBMS 

Participants 


Comparison 

Group 


Impact 


Postsecondary Enrollment (%) 


Any postsecondary institution 


91 


89 


1 ! 


89 


86 


4 ! 


87 


83 


4 ! 


Highest level of schooling attended 


Four-year college or university 


82 


72 


1 J ** 


74 


67 


7 ! 


72 


62 


10 ! 


Two-year college 


8 


16 


_g ** 


14 


19 


-5 ! 


16 


19 


-3 ! 


Vocational institution 


1 


3 


-2 ! 


2 


1 


1 ! 


0 


5 


-5 ! 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


26 


17 


9 ** 


23 


11 


*** 


34 


25 


9 


Less selective 


55 


54 


1 


50 


54 


-5 


38 


34 


4 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


88.5 


73.8 


14.8 ** 


86.2 


77.2 


9.0 


82.4 


52.6 


29 g *** 


Four-year colleges and universities 


83.3 


66.7 


16.7 ** 


75.9 


65.7 


10.2 


67.9 


41.3 


26.6 *** 


Two-year colleges and universities 


5.2 


7.1 


-1.9 


10.3 


11.4 


-1.2 


14.5 


11.4 


3.2 


Postsecondary Completion (%) 


Any degree, certificate, or license 


44 


41 


3 


43 


35 


o * 


46 


43 


3 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


35 


36 


-1 


35 


19 


16 *** # 


32 


25 


6 


Associate's degree 


4 


4 


1 ! 


8 


10 


-2 


7 


13 


-6 


Certificate or license 


5 


2 


3 ! 


1 


6 


-5 ! 


8 


3 


4 



Source: MPR analysis file "UBMSImpact6.sas." 

*/**/*** Impact estimate is statistically significant at the 0.10 / 0.05 / 0.01 level. 

# Impact estimate is significantly different from the impact on African Americans at the 0.10 level. 

! Statistical significance could not be assessed due to complete or semi-complete separation. 

Note 1 : All estimates are weighted to account for missing data. 

Note 2: The comparison group estimates and impact estimates are regression-adjusted (see Chapter III, Section A.4). 




APPENDIX D 



SAMPLE SIZES AND STANDARD ERRORS 




Table D.l 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.l 





Sample Size 


Standard Error 


Overall 


Total Credits 


1,677 


0.40 


Grade Point Average (GPA) 


1,677 


0.02 


Math Courses Taken 


Algebra (%) 


1,677 


0.70 


Geometry (%) 


1,677 


2.35 


Algebra 11 (%) 


1,677 


3.80 


Trigonometry (%) 


1,677 


4.16 


Analysis / Precalculus (%) 


1,677 


4.53 


Calculus (%) 


1,677 


2.77 


Advanced Placement Calculus (%) 


1,677 


1.43 


GPA in Algebra or above 


1,677 


0.05 


Science Courses Taken 


Biology (%) 


1,677 


0.91 


Chemistry (%) 


1,677 


1.61 


Physics (%) 


1,677 


3.07 


Advanced Placement Biology, Chemistry, or 


Physics (%) 


1,677 


1.38 


GPA in Biology, Chemistry, and Physics 


1,677 


0.04 


High School Status (%) 


Graduated 


1,677 


0.51 


Dropped out 


1,677 


0.37 


General Educational Development (GED) 


1,677 


0.38 



Source: MPR analysis file "UBMSlmpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series 
linearization methods. 
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Table D.2 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.2 





Participated in 


Regular Upward 


Did Not Participate 


in Regular 




Sample Size 


Standard Error 


Sample Size Standard Error 


Overall 


Total Credits 


640 


0.52 


1,037 


0.43 


Grade Point Average (GPA) 


640 


0.03 


1,037 


0.03 


Math Courses Taken 


Algebra (%) 


640 


1.10 


1,037 


0.63 


Geometry (%) 


640 


2.84 


1,037 


2.57 


Algebra II (%) 


640 


5.85 


1,037 


3.66 


Trigonometry (%) 


640 


6.08 


1,037 


4.59 


Analysis / Precalculus (%) 


640 


5.55 


1,037 


4.91 


Calculus (%) 


640 


4.21 


1,037 


3.17 


Advanced Placement Calculus (%) 


640 


2.55 


1,037 


1.69 


GPA in Algebra or above 


640 


0.08 


1,037 


0.05 


Science Courses Taken 


Biology (%) 


640 


1.05 


1,037 


1.05 


Chemistry (%) 


640 


2.44 


1,037 


1.88 


Physics (%) 


640 


5.27 


1,037 


3.43 


Advanced Placement Biology, Chemistry, or 


Physics (%) 


640 


1.44 


1,037 


1.66 


GPA in Biology, Chemistry, and Physics 


640 


0.06 


1,037 


0.05 


High School Status (%) 


Graduated 


640 


0.00 


1,037 


0.64 


Dropped out 


640 


0.00 


1,037 


0.47 


General Educational Development (GED) 


640 


0.00 


1,037 


0.49 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 





D-5 



Table D.3 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.3 





Male 




Female 






Sample Size Standard Error 


Sample Size Standard Error 


Overall 


Total Credits 


578 


0.47 


1,098 


0.47 


Grade Point Average (GPA) 


578 


0.04 


1,098 


0.03 


Math Courses Taken 


Algebra (%) 


578 


1.05 


1,098 


0.77 


Geometry (%) 


578 


2.91 


1,098 


2.49 


Algebra II (%) 


578 


4.40 


1,098 


4.43 


Trigonometry (%) 


578 


5.84 


1,098 


4.64 


Analysis / Precalculus (%) 


578 


4.87 


1,098 


5.10 


Calculus (%) 


578 


4.47 


1,098 


3.29 


Advanced Placement Calculus (%) 


578 


2.62 


1,098 


1.90 


GPA in Algebra or above 


578 


0.08 


1,098 


0.05 


Science Courses Taken 


Biology (%) 


578 


1.04 


1,098 


1.14 


Chemistry (%) 


578 


2.62 


1,098 


1.88 


Physics (%) 


578 


5.19 


1,098 


4.10 


Advanced Placement Biology, Chemistry, or 


Physics (%) 


578 


2.23 


1,098 


1.68 


GPA in Biology, Chemistry, and Physics 


578 


0.07 


1,098 


0.05 


High School Status (%) 


Graduated 


578 


0.84 


1,098 


0.66 


Dropped out 


578 


0.88 


1,098 


0.32 


General Educational Development (GED) 


578 


0.09 


1,098 


0.59 



Source: MPR analysis fde "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 
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Table D.4 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.4 





African American 


White 




Hispanic 






Sample Size 


Standard Error 


Sample Size Standard Error 


Sample Size Standard Error 


Overall 


Total Credits 


633 


0.55 


522 


0.82 


306 


0.92 


Grade Point Average (GPA) 


633 


0.04 


522 


0.03 


306 


0.05 


Math Courses Taken 


Algebra (%) 


633 


1.19 


522 


0.00 


306 


2.46 


Geometry (%) 


633 


2.82 


522 


2.07 


306 


5.01 


Algebra II (%) 


633 


5.78 


522 


5.67 


306 


5.66 


Trigonometry (%) 


633 


5.48 


522 


7.27 


306 


7.21 


Analysis / Precalculus (%) 


633 


6.47 


522 


6.71 


306 


7.97 


Calculus (%) 


633 


3.87 


522 


4.23 


306 


7.37 


Advanced Placement Calculus (%) 


633 


2.46 


522 


1.78 


306 


4.31 


GPA in Algebra or above 


633 


0.08 


522 


0.06 


306 


0.09 


Science Courses Taken 


Biology (%) 


633 


1.69 


522 


1.33 


306 


1.19 


Chemistry (%) 


633 


2.67 


522 


3.26 


306 


4.24 


Physics (%) 


633 


5.79 


522 


6.16 


306 


7.06 


Advanced Placement Biology, Chemistry, or 


Physics (%) 


633 


2.43 


522 


2.22 


306 


3.35 


GPA in Biology, Chemistry, and Physics 


633 


0.06 


522 


0.06 


306 


0.12 


High School Status (%) 


Graduated 


633 


0.00 


522 


1.09 


306 


1.46 


Dropped out 


633 


0.00 


522 


0.00 


306 


1.65 


General Educational Development (GED) 


633 


0.00 


522 


0.84 


306 


0.00 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 




Table D.5 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.5 





Sample Size 


Standard Error 


Postsecondary Enrollment (%) 


Any postsecondary institution 


1,438 


0.99 


Highest level of schooling attended 


Four-year college or university 


1,438 


2.28 


Two-year college 


1,438 


1.99 


Vocational institution 


1,438 


0.96 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


1,438 


3.44 


Less selective 


1,438 


4.05 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


1,158 


5.0 


Four-year colleges and universities 


1,158 


5.7 


Two-year colleges and universities 


1,158 


2.3 


Postsecondary Completion (%) 


Any degree, certificate, or license 


1,438 


3.54 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


1,438 


3.70 


Associate's degree 


1,438 


1.70 


Certificate or license 


1,438 


1.52 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization 
methods. 
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Table D.6 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.6 





Participated in 


Regular Upward 


Did Not Participate 


in Regular 




Sample Size 


Standard Error 


Sample Size Standard Error 


Postsecondary Enrollment (%) 


Any postsecondary institution 


552 


1.15 


886 


1.18 


Highest level of schooling attended 


Four-year college or university 


552 


4.54 


886 


2.47 


Two-year college 


552 


4.89 


886 


2.01 


V ocational institution 


552 


0.00 


886 


1.30 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


552 


5.56 


886 


3.79 


Less selective 


552 


5.88 


886 


4.49 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


432 


7.40 


726 


5.38 


Four-year colleges and universities 


432 


8.73 


726 


6.24 


Two-year colleges and universities 


432 


4.71 


726 


2.28 


Postsecondary Completion (%) 


Any degree, certificate, or license 


552 


5.88 


886 


3.86 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


552 


5.19 


886 


4.14 


Associate's degree 


552 


2.77 


886 


1.92 


Certificate or license 


552 


2.08 


886 


1.79 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 
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Table D.7 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.7 



Male Female 





Sample Size 


Standard Error 


Sample Size 


Standard Error 


Postsecondary Enrollment (%) 


Any postsecondary institution 


480 


1.62 


957 


1.31 


Highest level of schooling attended 


Four-year college or university 


480 


3.66 


957 


2.76 


Two-year college 


480 


3.80 


957 


2.29 


Vocational institution 


480 


1.11 


957 


1.32 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


480 


5.25 


957 


3.65 


Less selective 


480 


5.76 


957 


4.48 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


377 


7.80 


780 


4.85 


Four-year colleges and universities 


377 


8.97 


780 


5.47 


Two-year colleges and universities 


377 


4.73 


780 


2.43 


Postsecondary Completion (%) 


Any degree, certificate, or license 


480 


5.17 


957 


4.02 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


480 


4.70 


957 


3.97 


Associate's degree 


480 


3.35 


957 


1.74 


Certificate or license 


480 


2.43 


957 


1.74 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 
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Table D.8 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.8 



African American White Hispanic 





Sample Size 


Standard Error 


Sample Size 


Standard Error 


Sample Size 


Standard Error 


Postsecondary Enrollment (%) 


Any postsecondary institution 


523 


1.46 


468 


1.99 


261 


3.08 


Highest level of schooling attended 


Four-year college or university 


523 


3.28 


468 


4.52 


261 


3.87 


Two-year college 


523 


3.37 


468 


3.98 


261 


3.60 


Vocational institution 


523 


1.00 


468 


2.58 


261 


0.01 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


523 


5.41 


468 


5.92 


261 


7.61 


Less selective 


523 


6.33 


468 


6.07 


261 


7.95 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


405 


7.24 


394 


8.10 


211 


9.16 


Four-year colleges and universities 


405 


8.08 


394 


9.81 


211 


9.57 


Two-year colleges and universities 


405 


2.63 


394 


4.33 


211 


5.37 


Postsecondary Completion (%) 


Any degree, certificate, or license 


523 


6.30 


468 


5.01 


261 


6.62 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


523 


6.09 


468 


4.96 


261 


6.59 


Associate's degree 


523 


3.29 


468 


3.46 


261 


4.31 


Certificate or license 


523 


2.50 


468 


1.23 


261 


3.66 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 




Table D.9 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.9 





Sample Size 


Standard Error 


Field of Study at Most Recent PS Institution 

All PS institutions 


Math and science fields 


1,438 


3.73 


Social science fields 


1,438 


2.37 


Other fields 


1,438 


3.33 


Four-year colleges and universities 


Math and science fields 


1,438 


3.53 


Social science fields 


1,438 


2.37 


Other fields 


1,438 


3.23 


Earned Degree or Certificate in Field 

All PS institutions 


Math and science fields 


1,438 


2.84 


Social science fields 


1,438 


2.13 


Other fields 


1,438 


2.69 


Four-year colleges and universities 


Math and science fields 


1,438 


2.78 


Social science fields 


1,438 


2.17 


Other fields 


1,438 


2.56 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series 
linearization methods. 



D-l 1 







D-12 



Table D.10 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.10 





Participated in 


Regular Upward 


Did Not Participate 


in Regular 




Sample Size 


Standard Error 


Sample Size Standard Error 


Field of Study at Most Recent PS Institution 

All PS institutions 


Math and science fields 


552 


5.98 


886 


4.48 


Social science fields 


552 


4.70 


886 


2.64 


Other fields 


552 


6.10 


886 


4.01 


Four-year colleges and universities 


Math and science fields 


552 


5.76 


886 


4.21 


Social science fields 


552 


4.55 


886 


2.63 


Other fields 


552 


5.72 


886 


3.74 


Earned Degree or Certificate in Field 

All PS institutions 


Math and science fields 


552 


3.99 


886 


3.34 


Social science fields 


552 


2.90 


886 


2.49 


Other fields 


552 


4.59 


886 


3.15 


Four-year colleges and universities 


Math and science fields 


552 


2.89 


886 


3.59 


Social science fields 


552 


2.94 


886 


2.67 


Other fields 


552 


3.75 


886 


2.79 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 
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Table D. 11 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.l 1 





Male 




Female 






Sample Size Standard Error 


Sample Size Standard Error 


Field of Study at Most Recent PS Institution 

All PS institutions 


Math and science fields 


480 


6.16 


957 


4.14 


Social science fields 


480 


4.21 


957 


2.57 


Other fields 


480 


4.81 


957 


4.09 


Four-year colleges and universities 


Math and science fields 


480 


5.94 


957 


4.11 


Social science fields 


480 


3.72 


957 


2.60 


Other fields 


480 


4.55 


957 


3.76 


Earned Degree or Certificate in Field 

All PS institutions 


Math and science fields 


480 


3.87 


957 


3.50 


Social science fields 


480 


2.42 


957 


2.49 


Other fields 


480 


3.69 


957 


3.44 


Four-year colleges and universities 


Math and science fields 


480 


3.66 


957 


3.86 


Social science fields 


480 


2.18 


957 


2.66 


Other fields 


480 


2.84 


957 


3.01 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 
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Table D.12 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table B.12 



African American White Hispanic 





Sample Size 


Standard Error 


Sample Size 


Standard Error 


Sample Size 


Standard Error 


Field of Study at Most Recent PS Institution 

All PS institutions 


Math and science fields 


523 


5.84 


468 


6.00 


261 


7.31 


Social science fields 


523 


3.42 


468 


3.23 


261 


4.95 


Other fields 


523 


5.26 


468 


5.09 


261 


5.92 


Four-year colleges and universities 


Math and science fields 


523 


6.25 


468 


4.94 


261 


11.09 


Social science fields 


523 


3.63 


468 


3.18 


261 


3.70 


Other fields 


523 


4.68 


468 


4.48 


261 


5.58 


Earned Degree or Certificate in Field 

All PS institutions 


Math and science fields 


523 


4.64 


468 


5.17 


261 


7.69 


Social science fields 


523 


3.11 


468 


3.08 


261 


3.26 


Other fields 


523 


4.70 


468 


3.67 


261 


5.46 


Four-year colleges and universities 


Math and science fields 


523 


5.85 


468 


4.82 


261 


6.85 


Social science fields 


523 


3.06 


468 


3.08 


261 


3.65 


Other fields 


523 


4.10 


468 


3.66 


261 


4.73 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 



Table D. 13 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table C.l 





Sample Size 


Standard Error 


Postsecondary Enrollment (%) 


Any postsecondary institution 


1,438 


1.75 


Highest level of schooling attended 


Four-year college or university 


1,438 


2.54 


Two-year college 


1,438 


1.97 


Vocational institution 


1,438 


0.70 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


1,438 


3.34 


Less selective 


1,438 


3.78 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


1,438 


4.53 


Four-year colleges and universities 


1,438 


4.86 


Two-year colleges and universities 


1,438 


1.78 


Postsecondary Completion (%) 


Any degree, certificate, or license 


1,438 


3.50 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


1,438 


3.71 


Associate's degree 


1,438 


1.61 


Certificate or license 


1,438 


1.53 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series 
linearization methods. 
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Table D.14 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table C.2 





Participated in 


Regular Upward 


Did Not Participate 


in Regular 




Sample Size 


Standard Error 


Sample Size Standard Error 


Postsecondary Enrollment (%) 


Any postsecondary institution 


552 


3.46 


886 


1.92 


Highest level of schooling attended 


Four-year college or university 


552 


5.18 


886 


2.68 


Two-year college 


552 


4.89 


886 


2.07 


Vocational institution 


552 


0.00 


886 


0.92 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


552 


5.18 


886 


3.69 


Less selective 


552 


5.87 


886 


4.19 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


552 


6.67 


886 


4.89 


Four-year colleges and universities 


552 


7.39 


886 


5.36 


Two-year colleges and universities 


552 


3.36 


886 


1.90 


Postsecondary Completion (%) 


Any degree, certificate, or license 


552 


5.98 


886 


3.82 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


552 


4.97 


886 


4.11 


Associate's degree 


552 


2.77 


886 


1.83 


Certificate or license 


552 


2.02 


886 


1.90 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 
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Table D. 15 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table C.3 



Male Female 





Sample Size 


Standard Error 


Sample Size 


Standard Error 


Postsecondary Enrollment (%) 


Any postsecondary institution 


480 


3.07 


957 


2.12 


Highest level of schooling attended 


Four-year college or university 


480 


4.09 


957 


3.04 


Two-year college 


480 


3.90 


957 


2.09 


Vocational institution 


480 


0.81 


957 


0.87 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


480 


5.11 


957 


3.63 


Less selective 


480 


5.59 


957 


4.47 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


480 


6.71 


957 


5.12 


Four-year colleges and universities 


480 


7.10 


957 


5.21 


Two-year colleges and universities 


480 


2.95 


957 


2.26 


Postsecondary Completion (%) 


Any degree, certificate, or license 


480 


4.92 


957 


4.04 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


480 


4.57 


957 


4.03 


Associate's degree 


480 


2.87 


957 


1.75 


Certificate or license 


480 


2.21 


957 


1.98 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 
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Table D.16 

Sample Sizes and Standard Errors for Reported Impact Estimates: Table C.4 



African American White Hispanic 





Sample Size 


Standard Error 


Sample Size 


Standard Error 


Sample Size 


Standard Error 


Postsecondary Enrollment (%) 


Any postsecondary institution 


523 


2.64 


468 


3.43 


261 


4.57 


Highest level of schooling attended 


Four-year college or university 


523 


3.79 


468 


4.96 


261 


5.10 


Two-year college 


523 


2.98 


468 


4.74 


261 


4.41 


Vocational institution 


523 


0.96 


468 


1.52 


261 


0.00 


College Selectivity (%) 

Most selective four-year college or university 


More selective 


523 


4.93 


468 


5.85 


261 


7.88 


Less selective 


523 


5.90 


468 


6.13 


261 


7.49 


Postsecondary Credits Earned (mean) 


Two- and four-year colleges and universities 


523 


7.08 


468 


7.12 


261 


10.13 


Four-year colleges and universities 


523 


7.25 


468 


8.83 


261 


9.96 


Two-year colleges and universities 


523 


2.10 


468 


4.00 


261 


3.97 


Postsecondary Completion (%) 


Any degree, certificate, or license 


523 


6.33 


468 


4.84 


261 


6.69 


Highest degree, certificate, or license earned 


Bachelor's degree or higher 


523 


6.12 


468 


4.94 


261 


6.59 


Associate's degree 


523 


3.18 


468 


3.47 


261 


4.23 


Certificate or license 


523 


2.70 


468 


0.99 


261 


4.00 



Source: MPR analysis file "UBMSImpact6.sas." 

Note: Standard errors account for project clustering and were estimated using Taylor series linearization methods. 
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